OSGeo - User contributions [en]

A comprehensive list of webmapping toolkits

2011-09-15T09:28:17Z

Wiki-Cedricbr:

{| border="1" cellpadding="0" cellspacing="0"
|-
! Toolkit

! Description

! License

! Main map types

! Core languages
|-
| [http://www.cartoweb.org/ CartoWeb]

| CartoWeb is a comprehensive and ready-to-use Web-GIS (Geographical Information System) as well as a convenient framework for building advanced and customized applications.

| [http://www.gnu.org/copyleft/gpl.html GPL]

| [http://mapserver.org MapServer]

| [http://www.php.net PHP]
|-
| [http://chameleon.maptools.org/ Chameleon]

| Chameleon is a distributed, highly configurable, environment for developing Web Mapping applications. It is built on MapServer as the core mapping engine and works with all MapServer supported data formats through a regular MAP file.

| [http://chameleon.maptools.org/index.phtml?page=license.html Open Source]

| [http://mapserver.org MapServer], [http://www.opengeospatial.org/standards/wms WMS]

| [http://www.php.net PHP]
|-
| [http://maps.dnr.state.mn.us/tools/dbox/ dbox]

| dbox is really a collection of DHTML-based libraries for building highly interactive web-based mapping applications. The tools are meant to work directly with the MapServer web mapping system. They provide relatively autonomous functionality without restricting overall design. In fact, they were designed to be used with old fashioned elements like tables.

| Open Source

| [http://mapserver.org MapServer]

| [http://en.wikipedia.org/wiki/JavaScript JavaScript], [http://www.w3schools.com/dhtml/default.asp DHTML]
|-
| [http://surveillance.mcgill.ca/dracones/ Dracones]

| Dracones is a MapServer-based web mapping framework. Core components: A lightweight map widget, with a smooth navigation interface; Map layers with interactive behaviors, like mouse selection or tooltip (mouseover) information; Flexible query/extension mechanism; Handy other services like map image export, and history navigation

| [http://www.opensource.org/licenses/bsd-license.php BSD]

| [http://mapserver.org MapServer], [http://www.opengeospatial.org/standards/wms WMS]

| [http://www.php.net PHP], [http://www.python.org Python]
|-
| [http://trac.osgeo.org/fusion Fusion]

| Fusion is a web-mapping application development framework for MapGuide OS and MapServer built primarily in JavaScript. It allows non-spatial web developers to build rich mapping applications quickly and easily.

| [http://www.opensource.org/licenses/mit-license.php MIT]

| [http://mapserver.org MapServer], [http://mapguide.osgeo.org/ MapGuide OS]

| [http://www.php.net PHP]
|-
| [http://www.spatialytics.org/projects/geobiext/ GeoBIExt]

| a rich client visualization framework for SOLAP data, using GeoExt/OpenLayers as the web mapping front-end. It enables the creation of drillable and interactive geo-analytical dashboards or more static reports, which allow the analyze of data through different representations (maps, charts and cross-tabs) and levels of detail in order to take more informed and hence better decisions.

|  ?

| (see OpenLayers)

|  ?[http://www.java.com Java]
|-
| [http://geoext.org/ GeoExt]

| GeoExt brings together the geospatial know how of OpenLayers with the user interface savvy of Ext JS to help you build powerful desktop style GIS apps on the web with JavaScript.

| [http://www.opensource.org/licenses/bsd-license.php BSD]

| (see OpenLayers)

| [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://geojsp.org/ geojsp]

| geojsp is an open source (GPL) component that integrates geographic elements in your business intelligence infrastructure (geo-BI): thematic maps; flow maps; indicators; graphics (bar charts, pie chart, etc.)

| [http://www.gnu.org/copyleft/gpl.html GPL]

| (see OpenLayers)

| [http://www.java.com Java]
|-
| [http://www.spatialytics.org/projects/geomondrian/ GeoMondrian]

| GeoMondrian is a spatially-enabled version of Pentaho Analysis Services ([http://mondrian.pentaho.org/ Mondrian]). It provides a consistent integration of spatial objects into the OLAP data cube structure, instead of fetching them from a separate spatial database, web service or GIS file.

| [http://www.opensource.org/licenses/eclipse-1.0.php EPL]

| (see OpenLayers)

| [http://www.java.com Java]
|-
| [http://www.geomoose.org/ GeoMOOSE]

| GeoMOOSE is a Web Client Javascript Framework for displaying distributed cartographic data. GeoMOOSE has a number of strengths including modularity, configurability, and delivers a number of core functionalities in its packages.

| [http://www.opensource.org/licenses/mit-license.php MIT]

| [http://mapserver.org MapServer], (see OpenLayers)

| [http://en.wikipedia.org/wiki/JavaScript JavaScript], [http://www.php.net PHP]
|-
| [http://geoprisma.org/ GeoPrisma]

| ?

| Open Source

| (see OpenLayers)

|  ?
|-
| [http://code.google.com/p/geobi/ GeoReport]

| An interactive decision-support tool, providing a WebGIS OGC compliant interface, supported by analysis and reporting tools of external BI platforms. At present, GeoReport supports PentahoBI and SpagoBI.

| [http://www.gnu.org/licenses/lgpl.html LGPL]

| ?

|  ?
|-
| [https://sites.google.com/site/geoshieldproject/ GeoShield]

| GeoShield is a project born to offer a centralized way to define security access-control to geo-services. It acts like a proxy, intercepting all the communications between clients and OGC compliant services

| [http://www.opensource.org/licenses/bsd-license.php BSD]

| [http://geoserver.org/ GeoServer], [http://www.opengeospatial.org/standards/wms WMS], [http://www.opengeospatial.org/standards/wfs WFS], [http://www.opengeospatial.org/standards/wps WPS], [http://www.opengeospatial.org/standards/sos SOS]

| [http://www.java.com Java]
|-
| [http://geowebcache.org/ GeoWebCache]

| GeoWebCache is a Java web application used to cache map tiles coming from a variety of sources such as OGC Web Map Service (WMS). It implements various service interfaces (such as WMS-C, WMTS, TMS, Google Maps KML, Virtual Earth) in order to accelerate and optimize map image delivery. It can also recombine tiles to work with regular WMS clients.

| [http://www.gnu.org/licenses/lgpl.html LGPL]

| [http://www.opengeospatial.org/standards/wms WMS], [http://wiki.osgeo.org/wiki/WMS_Tile_Caching WMS-C], [http://wiki.osgeo.org/wiki/Tile_Map_Service_Specification TMS], [http://www.opengeospatial.org/standards/wmts WMTS]

| [http://www.java.com Java]
|-
| [http://mapas.mma.gov.br/i3geo/documentacao/files/principal-htm.html i3geo]

| i3geo provides a set of navigation tools, generation of analysis, sharing and generation of maps on demand.

| [http://www.gnu.org/copyleft/gpl.html GPL]

| [http://mapserver.org MapServer]

| [http://www.php.net PHP], [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://inteproxy.wald.intevation.org/ InteProxy]

| InteProxy is a tool that adds secure transmission via SSL as well as user authentication to those OWS-based applications that do not support these security features.

| [http://www.gnu.org/copyleft/gpl.html GPL]

| [http://www.opengeospatial.org/standards/wms WMS], [http://www.opengeospatial.org/standards/wfs WFS]

| [http://www.python.org Python]
|-
| [http://ka-map.maptools.org/ ka-Map]

| ka-Map is an open source project that is aimed at providing a javascript API for developing highly interactive web-mapping interfaces using features available in modern web browsers.

| [http://ka-map.maptools.org/index.phtml?page=license.html Open Source]

| [http://mapserver.org MapServer]

| [http://www.php.net PHP], [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://leaflet.cloudmade.com/ leaflet]

| Leaflet is a modern, lightweight JavaScript library for making tile-based interactive maps for both desktop and mobile web browsers.

| [http://www.opensource.org/licenses/bsd-license.php BSD]

| tiled layers, [http://www.opengeospatial.org/standards/wms WMS]

| [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://www.mapbender.org/ Mapbender]

| Mapbender is the back office software and client framework for spatial data infrastructures. It provides a data model and web based interfaces for displaying, navigating and querying OGC compliant map services.

| [http://www.gnu.org/copyleft/gpl.html GPL], [http://www.opensource.org/licenses/bsd-license.php BSD]

| [http://www.opengeospatial.org/standards/wms WMS], [http://www.opengeospatial.org/standards/wfs WFS]

| [http://www.php.net PHP], [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://docs.codehaus.org/display/MAP/Home MapBuilder]

| MapBuilder is a powerful, standards compliant geographic mapping client which runs in a web browser.

| [http://www.gnu.org/licenses/lgpl.html LGPL]

| [http://www.opengeospatial.org/standards/wms WMS], [http://www.opengeospatial.org/standards/wfs WFS]

| [http://en.wikipedia.org/wiki/JavaScript JavaScript]

|-
| [http://mapfaces.codehaus.org/ MapFaces]

| MapFaces is a JSF framework for mapping. This framework is implemented on top of GeotoolKit.org library, the reference implementation of OGC GeoAPI. Its main difference with other frameworks is the support of the on-the-fly reprojection for services like WMS, WMTS, TMS and others.

| [http://www.gnu.org/licenses/lgpl.html LGPL3]

| [http://www.opengeospatial.org/standards/wms WMS], [http://www.opengeospatial.org/standards/catalog CSW] + (all OpenLayers formats)

| [http://www.java.com/ Java], [http://en.wikipedia.org/wiki/JavaScript JavaScript]

|-
| [http://mapfish.org/ MapFish]

| MapFish is a flexible and complete framework for building rich web-mapping applications. It emphasizes high productivity, and high-quality development.

| [http://www.opensource.org/licenses/bsd-license.php BSD]

| [http://www.opengeospatial.org/standards/wms WMS], [http://www.opengeospatial.org/standards/wfs WFS]

| [http://www.python.org Python], [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://mapproxy.org/ MapProxy]

| It caches, accelerates and transforms data from existing map services and serves any desktop or web GIS client. MapProxy is a tile cache solution, but also offers many new and innovative features like full support for WMS clients.

| [http://www.apache.org/licenses/LICENSE-2.0.html ASL]

| [http://www.opengeospatial.org/standards/wms WMS], [http://wiki.osgeo.org/wiki/Tile_Map_Service_Specification TMS]

| [http://www.python.org Python]
|-
| [http://mapquery.org/ MapQuery]

| MapQuery is a jQuery plugin that you can use to add mapping to your website. Whether you quickly want to add a simple map to a page, or build a feature rich web application, MapQuery is just the thing you need.

| [http://www.opensource.org/licenses/mit-license.php MIT]

| (see OpenLayers)

| [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://datacrossing.crs4.it/en_Documentation_mscross.html msCross]

| msCross is a AJAX (WEB 2.0) WEB GIS client, initially developed to be a Javascript interface for UMN Mapserver. It was developed to allow users dinamically display geographical information layers on the web.

| [http://www.gnu.org/copyleft/gpl.html GPL]

| [http://mapserver.org MapServer]

| [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://www.openlayers.org/ OpenLayers]

| OpenLayers makes it easy to put a dynamic map in any web page. It can display map tiles and markers loaded from any source. OpenLayers has been developed to further the use of geographic information of all kinds.

| [http://svn.openlayers.org/trunk/openlayers/license.txt 2-clause BSD License]

| [http://resources.esri.com/help/9.3/arcgisserver/apis/rest/index.html ArcGIS Server], [http://www.opengeospatial.org/standards/gml GML], [http://code.google.com/apis/maps/index.html Google Maps], [http://code.google.com/apis/kml/documentation/ KML], [http://mapguide.osgeo.org/ MapGuide], [http://mapserver.org/ MapServer], [http://wiki.osgeo.org/wiki/Tile_Map_Service_Specification TMS], [http://www.microsoft.com/maps/developers/web.aspx Bing Maps], [http://www.opengeospatial.org/standards/wfs WFS], [http://www.opengeospatial.org/standards/wms WMS] and more!

| [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://openscales.org/ OpenScales]

| OpenScales is an open source (LGPL) mapping framework written in ActionScript 3 and Flex that enables developers to build Rich Internet Mapping Applications.

| [http://www.gnu.org/licenses/lgpl.html LGPL]

| [http://www.opengeospatial.org/standards/gml GML], [http://code.google.com/apis/kml/documentation/ KML], [http://www.openstreetmap.org/ OSM], [http://wiki.osgeo.org/wiki/Tile_Map_Service_Specification TMS], [http://www.opengeospatial.org/standards/wfs WFS], [http://www.opengeospatial.org/standards/wms WMS] and others.

| ActionScript, Flex, AIR
|-
| [http://www.pmapper.net/ p.mapper]

| The p.mapper framework is intended to offer broad functionality and multiple configurations in order to facilitate the setup of a MapServer application based on PHP/MapScript.

| [http://www.pmapper.net/license.shtml Open Source]

| [http://mapserver.org/ MapServer]

| [http://en.wikipedia.org/wiki/JavaScript JavaScript], [http://www.php.net PHP]
|-
| [http://polymaps.org/ Polymaps]

| Polymaps is a free JavaScript library for making dynamic, interactive maps in modern web browsers. It provides speedy display of multi-zoom datasets over maps, and supports a variety of visual presentations for tiled vector data, in addition to the usual cartography from OpenStreetMap, CloudMade, Bing, and other providers of image-based web maps.

| [https://github.com/simplegeo/polymaps/blob/master/LICENSE Open Source]

| [http://www.microsoft.com/maps/developers/web.aspx Bing Maps], [http://cloudmade.com/products/web-maps-api Cloudmade], [http://www.openstreetmap.org/ OSM]

| [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://www.spatialytics.org/projects/solaplayers/ SOLAPLayers]

| SOLAPLayers is a lightweight web cartographic component which enables navigation in SOLAP data cubes. It aims to be integrated into existing dashboard frameworks in order to produce interactive geo-analytical dashboards. 

| [http://www.opensource.org/licenses/bsd-license.php BSD], [http://www.opensource.org/licenses/eclipse-1.0.php EPL]

| (see OpenLayers)

| [http://www.java.com/ Java]
|-
| [http://www.spatialytics.org/projects/solaplayers/ tile5]

| Tile5 is an opensource HTML5 mapping javascript library that gives developers the ability to use existing mapping engines and provide a rich HTML5 experience. Tile5 can also be used to create more generic HTML5 tiling interfaces.

| [http://www.opensource.org/licenses/mit-license.php MIT]

| [http://www.microsoft.com/maps/developers/web.aspx Bing Maps], [http://cloudmade.com/products/web-maps-api Cloudmade], [http://www.openstreetmap.org/ OSM] and others.

| [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://tilecache.org/ TileCache]

| TileCache provides a Python-based [http://wiki.osgeo.org/index.php/WMS_Tiling_Client_Recommendation WMS-C]/TMS server, with pluggable caching mechanisms and rendering backends. In the simplest use case, TileCache requires only write access to a disk, the ability to run Python CGI scripts, and a WMS you want to be cached.

| [http://www.opensource.org/licenses/bsd-license.php BSD]

| [http://www.mapnik.org/ Mapnik], [http://mapserver.org MapServer], [http://www.opengeospatial.org/standards/wms WMS]

| [http://www.python.org/ Python]
|-
| [http://mapbox.com/tilemill/ TileMill]

| TileMill is an application for making beautiful maps. Whether you’re a journalist, web designer, researcher, or seasoned cartographer, TileMill is the design studio you need to create compelling, interactive maps.

| [http://www.opensource.org/licenses/bsd-license.php BSD]

| [http://www.mapnik.org/ Mapnik]

| CSS-like map styling language
|-
| [http://thematicmapping.org/engine/ Thematic Mapping Engine]

| TME enables you to visualise global statistics on Google Earth. The engine returns a KMZ file that you can open in Google Earth or download to your computer.

| [http://www.gnu.org/copyleft/gpl.html GPL]

| [http://code.google.com/apis/kml/documentation/ KML]

| [http://en.wikipedia.org/wiki/JavaScript JavaScript], [http://www.php.net PHP]
|}

A comprehensive list of webmapping toolkits

2011-09-15T09:25:51Z

Wiki-Cedricbr:

{| border="1" cellpadding="0" cellspacing="0"
|-
! Toolkit

! Description

! License

! Main map types

! Core languages
|-
| [http://www.cartoweb.org/ CartoWeb]

| CartoWeb is a comprehensive and ready-to-use Web-GIS (Geographical Information System) as well as a convenient framework for building advanced and customized applications.

| [http://www.gnu.org/copyleft/gpl.html GPL]

| [http://mapserver.org MapServer]

| [http://www.php.net PHP]
|-
| [http://chameleon.maptools.org/ Chameleon]

| Chameleon is a distributed, highly configurable, environment for developing Web Mapping applications. It is built on MapServer as the core mapping engine and works with all MapServer supported data formats through a regular MAP file.

| [http://chameleon.maptools.org/index.phtml?page=license.html Open Source]

| [http://mapserver.org MapServer], [http://www.opengeospatial.org/standards/wms WMS]

| [http://www.php.net PHP]
|-
| [http://maps.dnr.state.mn.us/tools/dbox/ dbox]

| dbox is really a collection of DHTML-based libraries for building highly interactive web-based mapping applications. The tools are meant to work directly with the MapServer web mapping system. They provide relatively autonomous functionality without restricting overall design. In fact, they were designed to be used with old fashioned elements like tables.

| Open Source

| [http://mapserver.org MapServer]

| [http://en.wikipedia.org/wiki/JavaScript JavaScript], [http://www.w3schools.com/dhtml/default.asp DHTML]
|-
| [http://surveillance.mcgill.ca/dracones/ Dracones]

| Dracones is a MapServer-based web mapping framework. Core components: A lightweight map widget, with a smooth navigation interface; Map layers with interactive behaviors, like mouse selection or tooltip (mouseover) information; Flexible query/extension mechanism; Handy other services like map image export, and history navigation

| [http://www.opensource.org/licenses/bsd-license.php BSD]

| [http://mapserver.org MapServer], [http://www.opengeospatial.org/standards/wms WMS]

| [http://www.php.net PHP], [http://www.python.org Python]
|-
| [http://trac.osgeo.org/fusion Fusion]

| Fusion is a web-mapping application development framework for MapGuide OS and MapServer built primarily in JavaScript. It allows non-spatial web developers to build rich mapping applications quickly and easily.

| [http://www.opensource.org/licenses/mit-license.php MIT]

| [http://mapserver.org MapServer], [http://mapguide.osgeo.org/ MapGuide OS]

| [http://www.php.net PHP]
|-
| [http://www.spatialytics.org/projects/geobiext/ GeoBIExt]

| a rich client visualization framework for SOLAP data, using GeoExt/OpenLayers as the web mapping front-end. It enables the creation of drillable and interactive geo-analytical dashboards or more static reports, which allow the analyze of data through different representations (maps, charts and cross-tabs) and levels of detail in order to take more informed and hence better decisions.

|  ?

| (see OpenLayers)

|  ?[http://www.java.com Java]
|-
| [http://geoext.org/ GeoExt]

| GeoExt brings together the geospatial know how of OpenLayers with the user interface savvy of Ext JS to help you build powerful desktop style GIS apps on the web with JavaScript.

| [http://www.opensource.org/licenses/bsd-license.php BSD]

| (see OpenLayers)

| [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://geojsp.org/ geojsp]

| geojsp is an open source (GPL) component that integrates geographic elements in your business intelligence infrastructure (geo-BI): thematic maps; flow maps; indicators; graphics (bar charts, pie chart, etc.)

| [http://www.gnu.org/copyleft/gpl.html GPL]

| (see OpenLayers)

| [http://www.java.com Java]
|-
| [http://www.spatialytics.org/projects/geomondrian/ GeoMondrian]

| GeoMondrian is a spatially-enabled version of Pentaho Analysis Services ([http://mondrian.pentaho.org/ Mondrian]). It provides a consistent integration of spatial objects into the OLAP data cube structure, instead of fetching them from a separate spatial database, web service or GIS file.

| [http://www.opensource.org/licenses/eclipse-1.0.php EPL]

| (see OpenLayers)

| [http://www.java.com Java]
|-
| [http://www.geomoose.org/ GeoMOOSE]

| GeoMOOSE is a Web Client Javascript Framework for displaying distributed cartographic data. GeoMOOSE has a number of strengths including modularity, configurability, and delivers a number of core functionalities in its packages.

| [http://www.opensource.org/licenses/mit-license.php MIT]

| [http://mapserver.org MapServer], (see OpenLayers)

| [http://en.wikipedia.org/wiki/JavaScript JavaScript], [http://www.php.net PHP]
|-
| [http://geoprisma.org/ GeoPrisma]

| ?

| Open Source

| (see OpenLayers)

|  ?
|-
| [http://code.google.com/p/geobi/ GeoReport]

| An interactive decision-support tool, providing a WebGIS OGC compliant interface, supported by analysis and reporting tools of external BI platforms. At present, GeoReport supports PentahoBI and SpagoBI.

| [http://www.gnu.org/licenses/lgpl.html LGPL]

| ?

|  ?
|-
| [https://sites.google.com/site/geoshieldproject/ GeoShield]

| GeoShield is a project born to offer a centralized way to define security access-control to geo-services. It acts like a proxy, intercepting all the communications between clients and OGC compliant services

| [http://www.opensource.org/licenses/bsd-license.php BSD]

| [http://geoserver.org/ GeoServer], [http://www.opengeospatial.org/standards/wms WMS], [http://www.opengeospatial.org/standards/wfs WFS], [http://www.opengeospatial.org/standards/wps WPS], [http://www.opengeospatial.org/standards/sos SOS]

| [http://www.java.com Java]
|-
| [http://geowebcache.org/ GeoWebCache]

| GeoWebCache is a Java web application used to cache map tiles coming from a variety of sources such as OGC Web Map Service (WMS). It implements various service interfaces (such as WMS-C, WMTS, TMS, Google Maps KML, Virtual Earth) in order to accelerate and optimize map image delivery. It can also recombine tiles to work with regular WMS clients.

| [http://www.gnu.org/licenses/lgpl.html LGPL]

| [http://www.opengeospatial.org/standards/wms WMS], [http://wiki.osgeo.org/wiki/WMS_Tile_Caching WMS-C], [http://wiki.osgeo.org/wiki/Tile_Map_Service_Specification TMS], [http://www.opengeospatial.org/standards/wmts WMTS]

| [http://www.java.com Java]
|-
| [http://mapas.mma.gov.br/i3geo/documentacao/files/principal-htm.html i3geo]

| i3geo provides a set of navigation tools, generation of analysis, sharing and generation of maps on demand.

| [http://www.gnu.org/copyleft/gpl.html GPL]

| [http://mapserver.org MapServer]

| [http://www.php.net PHP], [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://inteproxy.wald.intevation.org/ InteProxy]

| InteProxy is a tool that adds secure transmission via SSL as well as user authentication to those OWS-based applications that do not support these security features.

| [http://www.gnu.org/copyleft/gpl.html GPL]

| [http://www.opengeospatial.org/standards/wms WMS], [http://www.opengeospatial.org/standards/wfs WFS]

| [http://www.python.org Python]
|-
| [http://ka-map.maptools.org/ ka-Map]

| ka-Map is an open source project that is aimed at providing a javascript API for developing highly interactive web-mapping interfaces using features available in modern web browsers.

| [http://ka-map.maptools.org/index.phtml?page=license.html Open Source]

| [http://mapserver.org MapServer]

| [http://www.php.net PHP], [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://leaflet.cloudmade.com/ leaflet]

| Leaflet is a modern, lightweight JavaScript library for making tile-based interactive maps for both desktop and mobile web browsers.

| [http://www.opensource.org/licenses/bsd-license.php BSD]

| tiled layers, [http://www.opengeospatial.org/standards/wms WMS]

| [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://www.mapbender.org/ Mapbender]

| Mapbender is the back office software and client framework for spatial data infrastructures. It provides a data model and web based interfaces for displaying, navigating and querying OGC compliant map services.

| [http://www.gnu.org/copyleft/gpl.html GPL], [http://www.opensource.org/licenses/bsd-license.php BSD]

| [http://www.opengeospatial.org/standards/wms WMS], [http://www.opengeospatial.org/standards/wfs WFS]

| [http://www.php.net PHP], [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://docs.codehaus.org/display/MAP/Home MapBuilder]

| MapBuilder is a powerful, standards compliant geographic mapping client which runs in a web browser.

| [http://www.gnu.org/licenses/lgpl.html LGPL]

| [http://www.opengeospatial.org/standards/wms WMS], [http://www.opengeospatial.org/standards/wfs WFS]

| [http://en.wikipedia.org/wiki/JavaScript JavaScript]

|-
| [http://mapfaces.codehaus.org/ MapFaces]

| MapFaces is a JSF framework for mapping. This framework is implemented on top of GeotoolKit.org library, the reference implementation of OGC GeoAPI. Its main difference with other frameworks is the support of the on-the-fly reprojection for services like WMS, WMTS, TMS and others.

| [http://www.gnu.org/licenses/lgpl.html LGPL3]

| [http://www.opengeospatial.org/standards/wms WMS], [http://www.opengeospatial.org/standards/catalog CSW] + (all Openlayer formats)

| [http://www.java.com/ Java]

|-
| [http://mapfish.org/ MapFish]

| MapFish is a flexible and complete framework for building rich web-mapping applications. It emphasizes high productivity, and high-quality development.

| [http://www.opensource.org/licenses/bsd-license.php BSD]

| [http://www.opengeospatial.org/standards/wms WMS], [http://www.opengeospatial.org/standards/wfs WFS]

| [http://www.python.org Python], [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://mapproxy.org/ MapProxy]

| It caches, accelerates and transforms data from existing map services and serves any desktop or web GIS client. MapProxy is a tile cache solution, but also offers many new and innovative features like full support for WMS clients.

| [http://www.apache.org/licenses/LICENSE-2.0.html ASL]

| [http://www.opengeospatial.org/standards/wms WMS], [http://wiki.osgeo.org/wiki/Tile_Map_Service_Specification TMS]

| [http://www.python.org Python]
|-
| [http://mapquery.org/ MapQuery]

| MapQuery is a jQuery plugin that you can use to add mapping to your website. Whether you quickly want to add a simple map to a page, or build a feature rich web application, MapQuery is just the thing you need.

| [http://www.opensource.org/licenses/mit-license.php MIT]

| (see OpenLayers)

| [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://datacrossing.crs4.it/en_Documentation_mscross.html msCross]

| msCross is a AJAX (WEB 2.0) WEB GIS client, initially developed to be a Javascript interface for UMN Mapserver. It was developed to allow users dinamically display geographical information layers on the web.

| [http://www.gnu.org/copyleft/gpl.html GPL]

| [http://mapserver.org MapServer]

| [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://www.openlayers.org/ OpenLayers]

| OpenLayers makes it easy to put a dynamic map in any web page. It can display map tiles and markers loaded from any source. OpenLayers has been developed to further the use of geographic information of all kinds.

| [http://svn.openlayers.org/trunk/openlayers/license.txt 2-clause BSD License]

| [http://resources.esri.com/help/9.3/arcgisserver/apis/rest/index.html ArcGIS Server], [http://www.opengeospatial.org/standards/gml GML], [http://code.google.com/apis/maps/index.html Google Maps], [http://code.google.com/apis/kml/documentation/ KML], [http://mapguide.osgeo.org/ MapGuide], [http://mapserver.org/ MapServer], [http://wiki.osgeo.org/wiki/Tile_Map_Service_Specification TMS], [http://www.microsoft.com/maps/developers/web.aspx Bing Maps], [http://www.opengeospatial.org/standards/wfs WFS], [http://www.opengeospatial.org/standards/wms WMS] and more!

| [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://openscales.org/ OpenScales]

| OpenScales is an open source (LGPL) mapping framework written in ActionScript 3 and Flex that enables developers to build Rich Internet Mapping Applications.

| [http://www.gnu.org/licenses/lgpl.html LGPL]

| [http://www.opengeospatial.org/standards/gml GML], [http://code.google.com/apis/kml/documentation/ KML], [http://www.openstreetmap.org/ OSM], [http://wiki.osgeo.org/wiki/Tile_Map_Service_Specification TMS], [http://www.opengeospatial.org/standards/wfs WFS], [http://www.opengeospatial.org/standards/wms WMS] and others.

| ActionScript, Flex, AIR
|-
| [http://www.pmapper.net/ p.mapper]

| The p.mapper framework is intended to offer broad functionality and multiple configurations in order to facilitate the setup of a MapServer application based on PHP/MapScript.

| [http://www.pmapper.net/license.shtml Open Source]

| [http://mapserver.org/ MapServer]

| [http://en.wikipedia.org/wiki/JavaScript JavaScript], [http://www.php.net PHP]
|-
| [http://polymaps.org/ Polymaps]

| Polymaps is a free JavaScript library for making dynamic, interactive maps in modern web browsers. It provides speedy display of multi-zoom datasets over maps, and supports a variety of visual presentations for tiled vector data, in addition to the usual cartography from OpenStreetMap, CloudMade, Bing, and other providers of image-based web maps.

| [https://github.com/simplegeo/polymaps/blob/master/LICENSE Open Source]

| [http://www.microsoft.com/maps/developers/web.aspx Bing Maps], [http://cloudmade.com/products/web-maps-api Cloudmade], [http://www.openstreetmap.org/ OSM]

| [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://www.spatialytics.org/projects/solaplayers/ SOLAPLayers]

| SOLAPLayers is a lightweight web cartographic component which enables navigation in SOLAP data cubes. It aims to be integrated into existing dashboard frameworks in order to produce interactive geo-analytical dashboards. 

| [http://www.opensource.org/licenses/bsd-license.php BSD], [http://www.opensource.org/licenses/eclipse-1.0.php EPL]

| (see OpenLayers)

| [http://www.java.com/ Java]
|-
| [http://www.spatialytics.org/projects/solaplayers/ tile5]

| Tile5 is an opensource HTML5 mapping javascript library that gives developers the ability to use existing mapping engines and provide a rich HTML5 experience. Tile5 can also be used to create more generic HTML5 tiling interfaces.

| [http://www.opensource.org/licenses/mit-license.php MIT]

| [http://www.microsoft.com/maps/developers/web.aspx Bing Maps], [http://cloudmade.com/products/web-maps-api Cloudmade], [http://www.openstreetmap.org/ OSM] and others.

| [http://en.wikipedia.org/wiki/JavaScript JavaScript]
|-
| [http://tilecache.org/ TileCache]

| TileCache provides a Python-based [http://wiki.osgeo.org/index.php/WMS_Tiling_Client_Recommendation WMS-C]/TMS server, with pluggable caching mechanisms and rendering backends. In the simplest use case, TileCache requires only write access to a disk, the ability to run Python CGI scripts, and a WMS you want to be cached.

| [http://www.opensource.org/licenses/bsd-license.php BSD]

| [http://www.mapnik.org/ Mapnik], [http://mapserver.org MapServer], [http://www.opengeospatial.org/standards/wms WMS]

| [http://www.python.org/ Python]
|-
| [http://mapbox.com/tilemill/ TileMill]

| TileMill is an application for making beautiful maps. Whether you’re a journalist, web designer, researcher, or seasoned cartographer, TileMill is the design studio you need to create compelling, interactive maps.

| [http://www.opensource.org/licenses/bsd-license.php BSD]

| [http://www.mapnik.org/ Mapnik]

| CSS-like map styling language
|-
| [http://thematicmapping.org/engine/ Thematic Mapping Engine]

| TME enables you to visualise global statistics on Google Earth. The engine returns a KMZ file that you can open in Google Earth or download to your computer.

| [http://www.gnu.org/copyleft/gpl.html GPL]

| [http://code.google.com/apis/kml/documentation/ KML]

| [http://en.wikipedia.org/wiki/JavaScript JavaScript], [http://www.php.net PHP]
|}

Benchmarking 2010

2010-09-23T12:25:31Z

Wiki-Cedricbr: /* Constellation-SDI */

== Basic Premise ==

Following up on [[Benchmarking 2009|last year's exercise]], the performance shoot-out presentation at [[FOSS4G2010]] will test how long each Web mapping server takes to generate a map image, from a common set of spatial data, on a common platform. The data will be served by each Web mapping server through the WMS standard, which will serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to measure various aspects of those layers.

== Participants ==

{| border="1"
|-
| '''Mapping Server'''
| '''Development Team Leader'''
| '''IRC nick'''
| '''Server status'''
|-
| Cadcorp GeognoSIS
| [mailto:martin.daly--@--cadcorp.com Martin Daly]
| mpdaly
| Production release of server installed with preliminary configuration for SHP, PostGIS, TIF and ECW
|-
| Constellation-SDI
| [[User:Acuster|Adrian Custer]] [[User:Cedricbr|Cédric Briançon]]
| acuster cedricbr
| Running on port 8280. Installation done for Shapefiles and Rasters.
|-
| Erdas Apollo
| [mailto:dimitri.monie@erdas.com Dimitri Monie begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting]
| dmonie
| Install and configuration on-going on windows server, ecw WMS configuration done, shape and DB WMS configuration on going. Port 80, port 8080
|-
| GeoServer
| Andrea Aime
| aaime
| Normally running on port 8080 on the Linux server. Shapefiles and TIFF mosaic have been configured.
|-
| <s>MapGuide Open Source</s>
| <s>Zac Spitzer</s>
| <s>zacspitzer</s>
| ''inactive''
|-
| Mapnik
| [[User:Springmeyer|Dane Springmeyer]]
| springmeyer
| Paleoserver is running on port 8090(shape) / 8091(postgis) / 8092(raster - no reprojection). (mod_mapnik_wms is an alternate being tested, will run on same ports if active).
|-
| MapServer
| [[User:Jmckenna|Jeff McKenna]]
| jmckenna
| linux_wms_bm serving WMS requests for shps, on port 8081. windows wms_bm serving requests for shps, on port 8081.
|-
| Oracle MapViewer
| [[User:msmitherdc|Michael Smith]]
| msmith_
| Set up on port 8088
|-
| QGIS mapserver
| Marco Hugentobler
| marco___
| Set up on port 8081 (Linux server). Shapefiles and TIF (without reprojection) configured
|}

== Timeline ==

{| border="1"
|-
| January 1st, 2010
| begin contacting all mapping servers
|-
| June 1st, 2010
| commitment due by all interested mapping servers
|-
| June 2nd, 2010
| exercise begins (and weekly meetings)
|-
| August 1st, 2010
| final testing begins
|-
| August 23rd, 2010
| no further testing
|-
| September 6-9, 2010
| present results at FOSS4G2010
|}

== Rules of Engagement ==

# All parties must contribute any changes that they make to their software for this exercise, back to their community. Note that the changes don't have to be contributed before the conference, just in a reasonable period of time.
# Comparisons will be made of the best available version of the software, be it a formal release or a development version.
# Two tests will be run: one 'baseline' test with the data in its raw format (with spatial indexes), and another 'best effort' test where 'the sky is the limit' for what changes you want to make to the data (change format, generalize, etc)
# Teams must document all steps they did to manipulate the data/server for both the 'baseline' and 'best effort' tests. If a team does not document the steps on this wiki then that team's test results will not be used.
# Data formats to be used will be shapefiles for vectors, and uncompressed geotiffs for rasters.
# WMS output formats to be used will be png8 and png24 where possible

== Documenting Server Details and Differences ==

It is the responsibility of each team to document their setup with regard to data.

Please keep your notes in your servers directory in svn: http://svn.osgeo.org/osgeo/foss4g/benchmarking/

One particular thing that will differ is each team's use of spatial indexes on shapefiles.

Teams are '''not''' allowed to modify the .shp, .shx, .dbf or .prj files of the vector data (for the baseline benchmark). They can, however, create auxiliary files for spatial indexes and the like.

{| border="1"
| '''Mapping Server''' || '''ext''' || '''type''' || '''command to create'''
|-
| MapServer || .qix || quadtree || shptree <shpfile> ([[shptree notes]])
|-
| GeoServer || .qix || quadtree || Using the ones generated for MapServer atm
|-
| Mapnik || .index || quadtree || shapeindex <shpfile> ([[Benchmarking_2010/Mapnik_notes#shpindex_depth]])
|-
|Erdas || .rtr || RTree || RTreeBuilder basedir shapename max min ([[RTreeBuilder notes]])
|-
|Cadcorp || *.shp.idx || RTree || GUI command in desktop SIS ([[Benchmarking_2010/Cadcorp_notes]])
|-
| Constellation-SDI || .qix || quadtree || Using the ones generated by MapServer
|-
| MapViewer || .oix || RTree || GUI option in MapBuilder.jar, java map/theme/style builder
|}

* Note: in your server setup documentation be sure to record the EXACT command used to create the indexes, with appropriate options passed (e.g depth)

== Datasets ==

A set of data published by the spanish National Mapping Agency will be used. This data is free for non-commercial use, so it's perfectly OK to use it in the benchmark.

Vector data will be a topographic map, composed of several shapefiles (one shapefile per theme). Raster data will be 50cm/px aerial photography.

We still do not have the definitive datasets, due to their size and the processing time needed to put them together. You can get some sample data by reading [[Benchmarking 2010/How to get some sample data]].

== SVN ==

The project files (minus data) are stored in Subversion (http://svn.osgeo.org/osgeo/foss4g/benchmarking/).

== Hardware ==

Machine A (windows server)
* System Type: Dell PowerEdge R410
* Ship Date: 7/7/2010
* Processor: Intel® Xeon® E5630 2.53Ghz, 12M Cache,Turbo, HT, 1066MHz Max Mem
* 8GB Memory (4x2GB), 1333MHz Dual Ranked RDIMMs for 1Processor, Optimized
* 2TB 7.2K RPM SATA
* OS: Windows Server 64bit

Machine B (linux server)
* System Type: Dell PowerEdge R410
* Ship Date: 7/7/2010
* Processor: Intel® Xeon® E5630 2.53Ghz, 12M Cache,Turbo, HT, 1066MHz Max Mem
* 8GB Memory (4x2GB), 1333MHz Dual Ranked RDIMMs for 1Processor, Optimized
* 2TB 7.2K RPM SATA
* OS: Centos 5.5 x86-64

Machine C (DB Host)
* System Type: Gateway E6610D
* Processor, Intel Core2 Duo - E6750 2.66 Ghz
* 4Gb Ram
* 250Gb Hard Drive 7200 rpm
* OS: Centos 5.5 x86-64

Machine D (testing server (JMeter))
* System Type: Dell Precision Workstation 390
* Ship Date: 9/7/2006
* Processor, 6300, 1.86, 2M, Core Duo-conroe, Burn 2
* 2Gb RAM
* 160 Gb Hard drive 7200 rpm
* Service Tag: 5Q5LQB1
* OS: Centos 5.5 i386

== Communication ==

Coordination/communication is primarily via the Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking

Weekly meetings will occur through IRC chat in the #foss4g channel on irc.freenode.net

=== Next IRC meeting ===

* Wed August 18th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=08&day=18&hour=15&min=0&sec=0 15:00:00 UTC]
** Provisional Agenda:
*** baseline test styling equivalence: are we there yet?
*** starting and stopping all servers
*** slotting times for each team to run benchmarks
*** cleaning up the rendering benchmark blank tiles
*** graphic access to the JMeter machine
*** last testing date

=== Previous IRC meeting ===

* Wed August 11th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=08&day=11&hour=15&min=0&sec=0 15:00:00 UTC]
** Provisional Agenda:
*** Jmeter testing
**** what output formats will be requested?
**** what projections will be requested? (Platte-carré is not conformal)
**** what layer combinations will be requested? (All vector layers or various mixed subsets?)
**** what envelopes will be issued? (Frank's 2009 query generation script forces a hard cutoff at the largest scale)
**** what output metrics will be created?
**** how will the jmx file be designed? (need one .csv file per thread so all requests are different)
*** Status report from each team
*** Testing deadline: shall we extend it?
*** Can all server provide links to the sample requests and allow to build more? (style comparison, are we all doing the same work?)
*** Vector data
**** Changes in rendering (missing values for some attributes)
**** Gaps: what to do with the 9 missing sheets?
** Summary: [http://logs.qgis.org/foss4g/%23foss4g.2010-08-11.log log]

* Wed August 4th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=08&day=4&hour=15&min=0&sec=0 15:00:00 UTC]
** Provisional Agenda:
*** Machine status / issues
*** Data status / issues
**** Characters mis-encoded in shapefiles (e.g. catalan in labels): fixable? impact on PostGIS database?
**** Raster conversion to GeoTIFF (torrent of results?)
*** Styles status / issues
*** Jmeter testing
**** what output formats will be requested?
**** what projections will be requested? (Platte-carré is not conformal)
**** what layer combinations will be requested? (All vector layers or various mixed subset?)
**** what envelopes will be issued? (Frank's 2009 query generation script forces a hard cutoff at the largest scale)
**** what output metrics will be created?
**** how will the jmx file be designed? (need one .csv file per thread so all requests are different)
** Summary: [[Benchmarking 2010 IRC meeting logs Aug 4th]]

* Wed July 28th
(no info)

* Wed July 21st @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=07&day=21&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on data processing for all of Spain
*** update on server setup
*** update on data styling
*** checkin on server installs / library dependencies

* Wed July 14th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=07&day=14&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on data processing for all of Spain
*** update on server setup
*** data styling discussions
**** scales to use for layers
**** missing full road network
**** layer to use for labels along lines
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-07-14.log log])
*** Attendance:
**** Jeff McKenna (jmckenna)
**** Andrea Aime (aaime)
**** Iván Sánchez (IvanSanchez)
**** Michael Smith (msmith_)
**** Dimitri Monie (dmonie)
**** Dane Springmeyer (springmeyer)
*** Discussion:
**** data:
***** full dataset of Spain, vector and raster, has been processed by IvanSanchez, and he is checking it and will create a torrent for transfer, by the end of the week
***** because of shapefile size limit some layers may have to be split (contour-1.shp, contour-2.shp)
**** servers:
***** msmith has received both server machines
***** T1 router is being installed, it will be a dedicated line for this project
***** he is adding 2 TB drives to each
**** layer styling:
***** we notice that the Spain dataset we are using (BCN25) does not include street files (zoomed in road network)
***** we agreed to use contours for curved label tests instead
***** jmckenna will test the layer scales provided by Marco (QGIS mapserver)
***** aaime pointed out problem with the posted SLDs, in terms of the label property (jmckenna to look into it)
***** aaime pointed out that we should make sure that all servers compute the scales the same way
****** OGC recommends 90 DPI (GeoServer, Mapnik use the recommended 90...MapServer defaults to 72, but should be configurable)
****** all teams should look into how their engine handles this and report to group
**** other:
***** IvanSanchez reported that a Spanish custom-made WMS server may join the exercise (they access raster files only)
***** dmonie pointed out that Erdas has been quiet lately because one of the team leaders is on holiday

* Wed June 30th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=30&hour=15&min=0&sec=0 15:00:00 UTC]
** Summary: [http://logs.qgis.org/foss4g/%23foss4g.2010-06-30.log log]

* Wed June 30th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=30&hour=15&min=0&sec=0 15:00:00 UTC]
** Summary: [http://logs.qgis.org/foss4g/%23foss4g.2010-06-30.log log]

* Wed June 23rd @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=23&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on proposed layer styling
*** update on available server specs
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-06-23.log log])

* Wed June 16th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=16&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on server infrastructure
*** update on OSGeo-es dataset ([[Benchmarking 2010/How to get some sample data|Sample data download instructions]])
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-06-16.log log])
*** Attendance:
**** Jeff McKenna (jmckenna)
**** Andrea Aime (aaime)
**** Iván Sánchez (IvanSanchez)
**** Michael Smith (msmith_)
**** Cédric Briançon (cedricbr)
*** Discussion:
**** "base" run will use only shapefiles, "best" run will be any format (raster/vector)
**** teams can optionally test Oracle, PostGIS, Ingres connections during the "best" run
**** layers tested will be: labelled roads (with labels that follow the lines), thematic polygon map, and a test with labelled roads + thematic polygon + point layer in a single request (3 separate runs)
***** msmith and jmckenna to examine styling for one sheet (Barcelona, sheet# 420)
**** each base and best run will test data in raw projection (ETRS89 + UTM28/29/30/31) and then reprojected on-the-fly to the Google Mercator projection
**** threads (number of concurrent requests) will be: 1-2-4-8-16-32-64
**** US Army Corps is proposing to provide 4 desktop machines. 2 identical for testing ( win/lin ) and 1 for jmeter and 1 for databases
***** msmith will find out availability of 2-cpu machines this week

* Wed June 2nd @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=02&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** team representative introductions
*** Establish this year's 'Rules of Engagement' ([http://wiki.osgeo.org/wiki/Benchmarking_2009#Rules_of_Engagement previous RoE])
*** Discuss dataset to be used (including how to share the large dataset, for running local tests)
*** Discuss server infrastructure
*** Discuss possible changes to the test script parameters
*** Set next meeting date
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-06-02.log log])
*** Attendance:
**** Jeff McKenna, MapServer (jmckenna)
**** Cédric Briançon - Constellation (cedricbr)
**** Michael Smith - Oracle Mapviewer and Oracle connections in MapServer (msmith_)
**** Zac Spitzer - MapGuide (zacspitzer)
**** Martin Daly - Cadcorp (mpdaly)
**** Anne-Sophie Collignon - ERDAS (ascollignon)
**** Marco Hugentobler - QGIS mapserver (marco___)
**** Andrea Aime - GeoServer (aaime)
**** Dimitri Monie - ERDAS (dmonie)
**** Frank Warmerdam - MapServer, possibly Ingres as well (FrankW)
**** Iván Sánchez - provide geodata from Spanish sources and help out Mapnik (IvanSanchez)
**** Dane Springmeyer - Mapnik (springmeyer)
**** Johann Sorel - GeotoolKit / Constellation (Eclesia)
**** Adrian Custer - Constellation (acuster)
**** Pirmin Kalberer - QGIS mapserver (pirmin_k)
**** Trevor Wekel - MapGuide (trevorw)
**** Daniel Morissette - MapServer (danmo)
*** Discussion:
**** All agreed to this year's "Rules of Engagement"
***** All parties must contribute any changes that they make to their software for this exercise, back to their community. Note that the changes don't have to be contributed before the conference, just in a reasonable period of time.
***** Comparisons will be made of the best available version of the software, be it a formal release or a development version.
***** Two tests will be run: one 'baseline' test with the data in its raw format (with spatial indexes), and another 'best effort' test where 'the sky is the limit' for what changes you want to make to the data (change format, generalize, etc)
***** Teams must document all steps they did to manipulate the data/server for both the 'baseline' and 'best effort' tests. If a team does not document the steps on this wiki then that team's test results will not be used.
***** Data formats to be used will be shapefiles for vectors, and uncompressed geotiffs for rasters.
***** WMS output formats to be used will be png8 and png24 where possible
**** Dataset
***** Iván Sánchez Ortega from OSGeo-es is willing to provide local Spain datasets
***** data is for non-commercial use only
***** 10 GB of shapefiles
***** 100 GB of uncompressed rasters
**** Server infrastructure
***** possible hosts are msmith (US Army Corps) or Skygone (http://www.skygoneinc.com/)
***** need two identical servers, one running Unix and another Windows, plus a client machine
***** specs could be: quad core, 8GB of memory, 1TB hard drive
***** need fast pipe to server, as large datasets will be transferred
**** Next meeting date
***** all agreed to use same day and time next week, but limit the meeting length to one hour

== Sample Dataset Styling ==

* download [http://download.osgeo.org/benchmarking/barcelona-sample-dataset.zip sample dataset]
* layer [http://wiki.osgeo.org/wiki/Benchmarking_2010/How_to_get_some_sample_data#Vector_data_issues styling rules]
{| border="1"
|-
| '''Mapping Server'''
| '''GetMap'''
| '''SLD'''
|-
| MapServer
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 all layers]
| --
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=settlement&SRS=EPSG:25831&BBOX=426633,4579926,429740,4582528&FORMAT=image/png&WIDTH=950&HEIGHT=600 settlement]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/settlement-sld.xml settlement-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=building&SRS=EPSG:25831&BBOX=426633,4579926,429740,4582528&FORMAT=image/png&WIDTH=950&HEIGHT=600 building]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/building-sld.xml building-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=industry&SRS=EPSG:25831&BBOX=424932,4575961,427328,4577812&FORMAT=image/png&WIDTH=950&HEIGHT=600 industry]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/industry-sld.xml industry-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=contour_lines&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 contour lines]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/contour-sld.xml contour-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=motorway&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 motorway]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/motorway-sld.xml motorway-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=road&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 road]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/road-sld.xml road-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=ramp&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 ramp]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/ramp-sld.xml ramp-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=track&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 track]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/track-sld.xml track-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=point_geometry&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 point labels with geometry]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/point-geom-sld.xml point-geom-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=point_nogeom&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 point labels without geometry]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/point-nogeom-sld.xml point-nogeom-sld.xml]
|}

=== Scale dependent rules checking requests ===

These requests show the behavior of scale dependent styling around the critical scales.
The geographic area requested is always the same:

<tt>BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102</tt>

The image size changes instead to alter the scale:

* roughly 1:29k: <tt>WIDTH=824&HEIGHT=472</tt>
* roughly 1:31k: <tt>WIDTH=771&HEIGHT=442</tt>
* roughly 1:49k: <tt>WIDTH=488&HEIGHT=280</tt>
* roughly 1.51k: <tt>WIDTH=469&HEIGHT=269</tt>

The table allows to compare the various servers output. The GeoServer links use a vendor option to get the actual scale, as computed by the server, displayed at the bottom/right of the map.

{| border="1"
|-
| '''Server'''
| '''1:29k'''
| '''1:31k'''
| '''1:49k'''
| '''1:51k'''
|-
| Constellation-SDI
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
| GeoServer
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472&format_options=layout:scale 1:29k]
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442&format_options=layout:scale 1:31k]
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280&format_options=layout:scale 1:49k]
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269&format_options=layout:scale 1:51k]
|-
| Mapnik
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
| MapServer
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
| MapViewer
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
|-
| GeognoSIS
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
|-
| QGIS mapserver
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
|-
| ERDAS APOLLO
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472&srs=EPSG:4326&FORMAT=image/png 1:29k]
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442&srs=EPSG:4326&FORMAT=image/png 1:31k]
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280&height=600&srs=EPSG:4326&FORMAT=image/png 1:49k]
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269&srs=EPSG:4326&FORMAT=image/png 1:51k]
|}

== Benchmarking setup ==

Tests are going to be run using JMeter, using a progression of 1, 2, 4, 8, 16, 32, and 64 threads, each thread group doing 100, 200, 200, 400, 400, 800 requests respectively, for a total of 2200 requests.
The requests bounds and sizes are going to be picked up from a csv file generated by a random generator script.

The scripts to be run for the baseline tests, and repeated for the best effort one, are:
* raster data, JPEG output, EPSG:25831, scales between 1:1M and 1:10k
* raster data, JPEG output, EPSG:3857, scales between 1:1M and 1:10k
* vector data, PNG24 output, EPSG:4326, scales between 1:300k and 1:1k
* vector data, PNG24 output, EPSG:3875, scales between 1:300k and 1:1k
* raster + vector, JPEG output, EPSG:25831, scales between 1:300k and 1:1k
* raster + vector, JPEG output, EPSG:3785, scales between 1:300k and 1:1k

The areas to be tested are:
* 6 E, 38 N, -1 E, 43 N for vector tests
* 372360,4557880,484490,4631460 for raster tests

The raster layer in the baseline test is going to be the mosaic of GeoTIFF files Frank prepared out of the ECW files on the Linux server.

The vector data set is composed of a stack of individual layers to be stacked in the following order (bottom to top):
* settlements
* contour-0
* contour-1
* contour-2
* contour-3
* contour-4
* contour-5
* contour-6
* contour-7
* building
* industry
* track
* ramp
* road
* motorway
* point-labels-for-geometry
* point-labels-no-geometry

The best effort will use its own custom set of layers (e.g., it's likely contours will be merged into one, and so on)

== Live Benchmark WMS GetMap Requests ==

Note: sample bbox's fixed on Aug 12 to fit the proper aspect ratio of an 800/600 image request (springmeyer). Previous bbox for barcelona request was '1.8,41,2.4,42', new one used below is: '1.43333333333,41.0,2.76666666667,42.0'. Previous request for focused view was '2.1076723642349,41.407828508849,2.1178733654021,41.414271246429', new one is '2.10767236423,41.4072245022,2.1178733654,41.4148752531'. This was done to sidestep a deficiency in mapnik for affine transformation of geometries according to the WMS spec when the image w/h aspect does not match the bbox aspect, but was agreed upon as useful for these demo links by jeff and andrea in IRC.

=== MapServer ===

* Windows 64bit Server
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (zoomed-in)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (mid-zoom)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=316295,4519985.6640625,540555,4669784.3359375&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (whole-area)

* Linux
** [http://12.189.158.78:8081/cgi-bin/mapserv?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.78:8081/cgi-bin/mapserv?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev-label-fix] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev-label-fix] (labelled contours, larger view)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (zoomed-in)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (mid-zoom)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=316295,4519985.6640625,540555,4669784.3359375&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (whole-area)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle-raster-only.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg Oracle GeoRaster, MapServer 5.7-dev] (zoomed-in)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle-raster-only.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg Oracle GeoRaster, MapServer 5.7-dev] (mid-zoom)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle-raster-only.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=316295,4519985.6640625,540555,4669784.3359375&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg Oracle GeoRaster, MapServer 5.7-dev] (whole-area)

=== GeoServer ===

* [http://12.189.158.78:8080/geoserver/wms?SRS=EPSG%3A4326&LAYERS=shp_alllayers_label&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles GeoServer 2.1.x] (Barcelona extents)
*[http://12.189.158.78:8080/geoserver/wms?SRS=EPSG%3A4326&LAYERS=shp_alllayers_label&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Shapefiles GeoServer 2.1.x] (with labelled contours)
* [http://12.189.158.78:8080/geoserver/wms?EXCEPTIONS=application%2Fvnd.ogc.se_inimage&LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=2.10767236423,41.4072245022,2.1178733654,41.41487525316&WIDTH=800&HEIGHT=600 Shapefiles GeoServer 2.1.x] (labelled contours, larger view)
* [http://12.189.158.78:8080/geoserver/wms?LAYERS=ras%3Amosaic_tif&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 TIFF mosaic, zoomed in, GeoServer 2.1.x]
* [http://12.189.158.78:8080/geoserver/wms?LAYERS=ras%3Amosaic_tif&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 TIFF mosaic, mid zoom, GeoServer 2.1.x]
* [http://12.189.158.78:8080/geoserver/wms?LAYERS=ras%3Amosaic_tif&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 TIFF mosaic, whole area, GeoServer 2.1.x]

=== Mapnik ===
* [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles Mapnik2 / paleoserver] (Barcelona extents)
* [http://12.189.158.78:8091/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles Mapnik0.7.2 / mod_mapnik_wms] (Barcelona extents)
* [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Shapefiles Mapnik2 / paleoserver] (larger view - labeled contours)

=== Constellation-SDI ===
* [http://12.189.158.78:8280/constellation/WS/wms?SRS=EPSG%3A4326&LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles Constellation] (Barcelona extents)
* [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=2.10767236423,41.4072245022,2.1178733654,41.41487525316&WIDTH=800&HEIGHT=600 Shapefiles Constellation] (labelled contours, larger view)
* [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=bench:orthophoto&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=800&HEIGHT=600 GeoTIFF Constellation] (zoomed-in)
* [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=bench:orthophoto&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600 GeoTIFF Constellation] (mid-zoom)
* [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=bench:orthophoto&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&width=800&height=600 GeoTIFF Constellation] (whole-area)

=== Oracle MapViewer ===
*[http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600&srs=EPSG:4326&FORMAT=image/png&aa=TRUE Oracle DB / MapViewer] (Barcelona Extents)

*[http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png&aa=TRUE Oracle DB / MapViewer] (with labeled contours)

=== Cadcorp GeognoSIS ===
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600 Shapefiles, GeognoSIS 7.0] (Barcelona Extents)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Shapefiles, GeognoSIS 7.0] (with labelled contours)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-postgis&BGCOLOR=0xFFFFFF&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600 PostGIS, GeognoSIS 7.0] (Barcelona Extents)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-postgis&BGCOLOR=0xFFFFFF&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 PostGIS, GeognoSIS 7.0] (with labelled contours)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-oracle&BGCOLOR=0xFFFFFF&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600 Oracle, GeognoSIS 7.0] (Barcelona Extents)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-oracle&BGCOLOR=0xFFFFFF&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Oracle, GeognoSIS 7.0] ('''no contours''' - under investigation)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-tif&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 TIFF mosaic, zoomed in, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-tif&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 TIFF mosaic, mid zoom, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-tif&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 TIFF mosaic, whole area, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-ecw&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 ECW, zoomed in, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-ecw&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 ECW, mid zoom, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-ecw&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 ECW, whole area, GeognoSIS 7.0]

=== QGIS mapserver ===
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SRS=EPSG:4326&LAYERS=shp_vector&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles, QGIS mapserver] (Barcelona Extents)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, QGIS mapserver] (with labeled contours)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/pg.qgs&SRS=EPSG:4326&LAYERS=shp_vector&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 PostGIS, QGIS mapserver] (Barcelona Extents)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/pg.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png PostGIS, QGIS mapserver] (with labeled contours)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector_raster.qgs&LAYERS=vrt&STYLES=&FORMAT=image/jpg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3043&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 TIFF mosaic, zoomed in, QGIS mapserver]
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector_raster.qgs&LAYERS=vrt&STYLES=&FORMAT=image/jpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3043&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 TIFF mosaic, mid zoom, QGIS mapserver]
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector_raster.qgs&LAYERS=vrt&STYLES=&FORMAT=image/jpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3043&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 TIFF mosaic, whole area, QGIS mapserver]

=== ERDAS APOLLO ===

Windows 64bit Server
* [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, APOLLO 10.1.2] (Barcelona extents)
* [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, APOLLO 10.1.2] (with labelled contours)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=TIFF&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 Geotiff, APOLLO 10.1.2] (Zoomed-in)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=TIFF&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 Geotiff, APOLLO 10.1.2](mid-zoom)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=TIFF&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 Geotiff, APOLLO 10.1.2] (Whole Area)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=BENCHMARK&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 ECW, APOLLO 10.1.2] (Zoomed-in)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=BENCHMARK&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 ECW, APOLLO 10.1.2] (mid-zoom)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=BENCHMARK&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 ECW, APOLLO 10.1.2] (Whole Area)

== Starting and stopping the various servers ==

* GeoServer on the Linux box:
** <tt>cd /benchmarking/geoserver</tt>
** starting: <tt>/benchmarking/geoserver/start_geoserver.sh</tt>
** stopping gracefully: <tt>/benchmarking/geoserver/stop_geoserver.sh</tt>
** check it's really dead: <tt>ps aux | grep GEOSERVER</tt> . If it hasn't stopped gracefully use <tt>kill -9 <pid></tt> to terminate the process

* MapServer and QGIS Mapserver on the Linux box (both are running in the same Apache instance):
** starting: /opt/mapserver/bin/apachectl start
** stopping: /opt/mapserver/bin/apachectl stop

* MapServer on Windows box:
** Command prompt (right-click "Run as Administrator")
** cd \ms4w
** apache-stop.bat
** apache-start.bat

* Mapnik:
** paleoserver (standalone daemon):
*** stopping: /opt/mapnik/paleoserver_stop.sh
*** starting: /opt/mapnik/paleoserver_start.sh
** mod_mapnik_wms (runs within apache):
** starting: /opt/mapnik/mod_mapnik_start.sh
** stopping: /opt/mapnik/mod_mapnik_stop.sh

* Mapviewer:
** <tt>cd /benchmarking/mapviewer</tt>
** starting: <tt> /opt/mapviewer/start_mv.sh </tt>
** stopping: <tt> /opt/mapviewer/stop_mv.sh </tt>
** check it's really dead: <tt>ps aux | grep java | grep mapviewer</tt> . If it hasn't stopped gracefully use <tt>kill -9 <pid></tt> to terminate the process

* Cadcorp GeognoSIS
** 7.0
*** Command line (with "Run as Administrator")
**** starting: net start geognosis-70
**** stopping: net stop geognosis-70
*** Or use services.msc, service name geognosis-70
** 7.1
*** Same as 7.0, but with "-71" suffix

* Erdas Windows Server
** APOLLO SDI (vector services)
*** Command Line (with "Run as Administrator")
**** starting : cd C:\opt\ERDAS\APOLLOSDI\tomcat\bin\startup.bat
**** stopping : cd C:\opt\ERDAS\APOLLOSDI\tomcat\bin\shutdown.bat
*** check the java process for erdas user has been killed
** APOLLO IWS (raster services)
*** Open IIS (Control Panel\Administrative Tools --> Internet Information Services (IIS) Manager)
**** starting : select under "Site" "Default Web Site" --> Manage Web Site : Start
**** Stopping : select under "Site" "Default Web Site" --> Manage Web Site : Stop

* Constellation on the Linux box (you must be logged in as constellation user, do not launch it as root !):
** starting: <tt>/opt/constellation/apache-tomcat-6.0.28/bin/startup.sh</tt>
** stopping: <tt>/opt/constellation/apache-tomcat-6.0.28/bin/shutdown.sh</tt>
** check it's really dead: <tt>ps aux | grep constellation</tt> . If it hasn't stopped gracefully use <tt>kill -9 <pid></tt> to terminate the process

* Oracle Database
** Login to database machine and sudo to user oracle
***Starting: lsntctl start; dbstart
***Stopping: lsntcrl stop: dbshut

== List of Software Enhancements ==

=== GeoServer ===

* support DBF larger than 2GB
* avoid reading the Z dimension when flat geometries are all that's needed
* generally improve shapefile reading: don't open the shp and dbf file if the spatial index says there is no data to read, bbox based feature skipping during filtering, limit number of disk reads to the bare minimum
* perform fast rectangular clipping of geometries before sending them to the renderer
* direct rendering path for raster to avoid the java2d bottlenecks
* improved .qix index file generation based on node load factor instead of tree depth, node collapsing for degenerate linear chains
* light parallelization of the renderer so that feature reading and rendering can be performed in parallel
* better metadata caching (feature type structure and the like)

=== MapServer ===
* large DBF support ([https://trac.osgeo.org/mapserver/ticket/3514 ticket])
* improving labels on curved lines ([http://trac.osgeo.org/mapserver/ticket/3523 ticket])
* handling label precision ([http://trac.osgeo.org/mapserver/ticket/2950 ticket])
* improving Oracle GeoRaster GDAL driver ([http://trac.osgeo.org/gdal/ticket/3746 ticket])

=== Constellation-SDI ===

Detailed in the [[Benchmarking_2010/Constellation-SDI|Constellation-SDI]] page.

=== Cadcorp GeognoSIS ===

Detailed in the [[Benchmarking_2010/Cadcorp_notes|Cadcorp notes]] page.

=== ERDAS APOLLO ===

For APOLLO IWS :
* Upgrade to GDAL 1.7.2
* Direct access to GDAL for Geotiff reader
* Improved the mosaicing of TIF datasets to remove seam lines
For APOLLO SDI :
* Tomcat 6 and Java 6: improved performances over Tomcat 5.5 and Java 5 Rendering :
* Clash management for contour labels
* Multipath rendering optimisation
* Low level code optimisation
* New option for shape index in memory

=== Mapnik ===
Detailed at: [[Benchmarking_2010/Mapnik_notes#Improvements]]

== List of Performance Results ==

=== Constellation-SDI ===

Detailed in the [[Benchmarking_2010/Constellation-SDI|Constellation-SDI]] page.

== External Related Links ==

*[[FOSS4G Benchmark|FOSS4G WMS Benchmark]]
*[http://sourceforge.net/projects/wmstester/ WMSTester - tool for testing not from OSGeo]

----

[[Category:FOSS4G2010]] [[Category:FOSS4G]]

Benchmarking 2010

2010-09-23T12:21:55Z

Wiki-Cedricbr: /* Constellation-SDI */

== Basic Premise ==

Following up on [[Benchmarking 2009|last year's exercise]], the performance shoot-out presentation at [[FOSS4G2010]] will test how long each Web mapping server takes to generate a map image, from a common set of spatial data, on a common platform. The data will be served by each Web mapping server through the WMS standard, which will serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to measure various aspects of those layers.

== Participants ==

{| border="1"
|-
| '''Mapping Server'''
| '''Development Team Leader'''
| '''IRC nick'''
| '''Server status'''
|-
| Cadcorp GeognoSIS
| [mailto:martin.daly--@--cadcorp.com Martin Daly]
| mpdaly
| Production release of server installed with preliminary configuration for SHP, PostGIS, TIF and ECW
|-
| Constellation-SDI
| [[User:Acuster|Adrian Custer]] [[User:Cedricbr|Cédric Briançon]]
| acuster cedricbr
| Running on port 8280. Installation done for Shapefiles and Rasters.
|-
| Erdas Apollo
| [mailto:dimitri.monie@erdas.com Dimitri Monie begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting]
| dmonie
| Install and configuration on-going on windows server, ecw WMS configuration done, shape and DB WMS configuration on going. Port 80, port 8080
|-
| GeoServer
| Andrea Aime
| aaime
| Normally running on port 8080 on the Linux server. Shapefiles and TIFF mosaic have been configured.
|-
| <s>MapGuide Open Source</s>
| <s>Zac Spitzer</s>
| <s>zacspitzer</s>
| ''inactive''
|-
| Mapnik
| [[User:Springmeyer|Dane Springmeyer]]
| springmeyer
| Paleoserver is running on port 8090(shape) / 8091(postgis) / 8092(raster - no reprojection). (mod_mapnik_wms is an alternate being tested, will run on same ports if active).
|-
| MapServer
| [[User:Jmckenna|Jeff McKenna]]
| jmckenna
| linux_wms_bm serving WMS requests for shps, on port 8081. windows wms_bm serving requests for shps, on port 8081.
|-
| Oracle MapViewer
| [[User:msmitherdc|Michael Smith]]
| msmith_
| Set up on port 8088
|-
| QGIS mapserver
| Marco Hugentobler
| marco___
| Set up on port 8081 (Linux server). Shapefiles and TIF (without reprojection) configured
|}

== Timeline ==

{| border="1"
|-
| January 1st, 2010
| begin contacting all mapping servers
|-
| June 1st, 2010
| commitment due by all interested mapping servers
|-
| June 2nd, 2010
| exercise begins (and weekly meetings)
|-
| August 1st, 2010
| final testing begins
|-
| August 23rd, 2010
| no further testing
|-
| September 6-9, 2010
| present results at FOSS4G2010
|}

== Rules of Engagement ==

# All parties must contribute any changes that they make to their software for this exercise, back to their community. Note that the changes don't have to be contributed before the conference, just in a reasonable period of time.
# Comparisons will be made of the best available version of the software, be it a formal release or a development version.
# Two tests will be run: one 'baseline' test with the data in its raw format (with spatial indexes), and another 'best effort' test where 'the sky is the limit' for what changes you want to make to the data (change format, generalize, etc)
# Teams must document all steps they did to manipulate the data/server for both the 'baseline' and 'best effort' tests. If a team does not document the steps on this wiki then that team's test results will not be used.
# Data formats to be used will be shapefiles for vectors, and uncompressed geotiffs for rasters.
# WMS output formats to be used will be png8 and png24 where possible

== Documenting Server Details and Differences ==

It is the responsibility of each team to document their setup with regard to data.

Please keep your notes in your servers directory in svn: http://svn.osgeo.org/osgeo/foss4g/benchmarking/

One particular thing that will differ is each team's use of spatial indexes on shapefiles.

Teams are '''not''' allowed to modify the .shp, .shx, .dbf or .prj files of the vector data (for the baseline benchmark). They can, however, create auxiliary files for spatial indexes and the like.

{| border="1"
| '''Mapping Server''' || '''ext''' || '''type''' || '''command to create'''
|-
| MapServer || .qix || quadtree || shptree <shpfile> ([[shptree notes]])
|-
| GeoServer || .qix || quadtree || Using the ones generated for MapServer atm
|-
| Mapnik || .index || quadtree || shapeindex <shpfile> ([[Benchmarking_2010/Mapnik_notes#shpindex_depth]])
|-
|Erdas || .rtr || RTree || RTreeBuilder basedir shapename max min ([[RTreeBuilder notes]])
|-
|Cadcorp || *.shp.idx || RTree || GUI command in desktop SIS ([[Benchmarking_2010/Cadcorp_notes]])
|-
| Constellation-SDI || .qix || quadtree || Using the ones generated by MapServer
|-
| MapViewer || .oix || RTree || GUI option in MapBuilder.jar, java map/theme/style builder
|}

* Note: in your server setup documentation be sure to record the EXACT command used to create the indexes, with appropriate options passed (e.g depth)

== Datasets ==

A set of data published by the spanish National Mapping Agency will be used. This data is free for non-commercial use, so it's perfectly OK to use it in the benchmark.

Vector data will be a topographic map, composed of several shapefiles (one shapefile per theme). Raster data will be 50cm/px aerial photography.

We still do not have the definitive datasets, due to their size and the processing time needed to put them together. You can get some sample data by reading [[Benchmarking 2010/How to get some sample data]].

== SVN ==

The project files (minus data) are stored in Subversion (http://svn.osgeo.org/osgeo/foss4g/benchmarking/).

== Hardware ==

Machine A (windows server)
* System Type: Dell PowerEdge R410
* Ship Date: 7/7/2010
* Processor: Intel® Xeon® E5630 2.53Ghz, 12M Cache,Turbo, HT, 1066MHz Max Mem
* 8GB Memory (4x2GB), 1333MHz Dual Ranked RDIMMs for 1Processor, Optimized
* 2TB 7.2K RPM SATA
* OS: Windows Server 64bit

Machine B (linux server)
* System Type: Dell PowerEdge R410
* Ship Date: 7/7/2010
* Processor: Intel® Xeon® E5630 2.53Ghz, 12M Cache,Turbo, HT, 1066MHz Max Mem
* 8GB Memory (4x2GB), 1333MHz Dual Ranked RDIMMs for 1Processor, Optimized
* 2TB 7.2K RPM SATA
* OS: Centos 5.5 x86-64

Machine C (DB Host)
* System Type: Gateway E6610D
* Processor, Intel Core2 Duo - E6750 2.66 Ghz
* 4Gb Ram
* 250Gb Hard Drive 7200 rpm
* OS: Centos 5.5 x86-64

Machine D (testing server (JMeter))
* System Type: Dell Precision Workstation 390
* Ship Date: 9/7/2006
* Processor, 6300, 1.86, 2M, Core Duo-conroe, Burn 2
* 2Gb RAM
* 160 Gb Hard drive 7200 rpm
* Service Tag: 5Q5LQB1
* OS: Centos 5.5 i386

== Communication ==

Coordination/communication is primarily via the Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking

Weekly meetings will occur through IRC chat in the #foss4g channel on irc.freenode.net

=== Next IRC meeting ===

* Wed August 18th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=08&day=18&hour=15&min=0&sec=0 15:00:00 UTC]
** Provisional Agenda:
*** baseline test styling equivalence: are we there yet?
*** starting and stopping all servers
*** slotting times for each team to run benchmarks
*** cleaning up the rendering benchmark blank tiles
*** graphic access to the JMeter machine
*** last testing date

=== Previous IRC meeting ===

* Wed August 11th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=08&day=11&hour=15&min=0&sec=0 15:00:00 UTC]
** Provisional Agenda:
*** Jmeter testing
**** what output formats will be requested?
**** what projections will be requested? (Platte-carré is not conformal)
**** what layer combinations will be requested? (All vector layers or various mixed subsets?)
**** what envelopes will be issued? (Frank's 2009 query generation script forces a hard cutoff at the largest scale)
**** what output metrics will be created?
**** how will the jmx file be designed? (need one .csv file per thread so all requests are different)
*** Status report from each team
*** Testing deadline: shall we extend it?
*** Can all server provide links to the sample requests and allow to build more? (style comparison, are we all doing the same work?)
*** Vector data
**** Changes in rendering (missing values for some attributes)
**** Gaps: what to do with the 9 missing sheets?
** Summary: [http://logs.qgis.org/foss4g/%23foss4g.2010-08-11.log log]

* Wed August 4th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=08&day=4&hour=15&min=0&sec=0 15:00:00 UTC]
** Provisional Agenda:
*** Machine status / issues
*** Data status / issues
**** Characters mis-encoded in shapefiles (e.g. catalan in labels): fixable? impact on PostGIS database?
**** Raster conversion to GeoTIFF (torrent of results?)
*** Styles status / issues
*** Jmeter testing
**** what output formats will be requested?
**** what projections will be requested? (Platte-carré is not conformal)
**** what layer combinations will be requested? (All vector layers or various mixed subset?)
**** what envelopes will be issued? (Frank's 2009 query generation script forces a hard cutoff at the largest scale)
**** what output metrics will be created?
**** how will the jmx file be designed? (need one .csv file per thread so all requests are different)
** Summary: [[Benchmarking 2010 IRC meeting logs Aug 4th]]

* Wed July 28th
(no info)

* Wed July 21st @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=07&day=21&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on data processing for all of Spain
*** update on server setup
*** update on data styling
*** checkin on server installs / library dependencies

* Wed July 14th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=07&day=14&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on data processing for all of Spain
*** update on server setup
*** data styling discussions
**** scales to use for layers
**** missing full road network
**** layer to use for labels along lines
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-07-14.log log])
*** Attendance:
**** Jeff McKenna (jmckenna)
**** Andrea Aime (aaime)
**** Iván Sánchez (IvanSanchez)
**** Michael Smith (msmith_)
**** Dimitri Monie (dmonie)
**** Dane Springmeyer (springmeyer)
*** Discussion:
**** data:
***** full dataset of Spain, vector and raster, has been processed by IvanSanchez, and he is checking it and will create a torrent for transfer, by the end of the week
***** because of shapefile size limit some layers may have to be split (contour-1.shp, contour-2.shp)
**** servers:
***** msmith has received both server machines
***** T1 router is being installed, it will be a dedicated line for this project
***** he is adding 2 TB drives to each
**** layer styling:
***** we notice that the Spain dataset we are using (BCN25) does not include street files (zoomed in road network)
***** we agreed to use contours for curved label tests instead
***** jmckenna will test the layer scales provided by Marco (QGIS mapserver)
***** aaime pointed out problem with the posted SLDs, in terms of the label property (jmckenna to look into it)
***** aaime pointed out that we should make sure that all servers compute the scales the same way
****** OGC recommends 90 DPI (GeoServer, Mapnik use the recommended 90...MapServer defaults to 72, but should be configurable)
****** all teams should look into how their engine handles this and report to group
**** other:
***** IvanSanchez reported that a Spanish custom-made WMS server may join the exercise (they access raster files only)
***** dmonie pointed out that Erdas has been quiet lately because one of the team leaders is on holiday

* Wed June 30th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=30&hour=15&min=0&sec=0 15:00:00 UTC]
** Summary: [http://logs.qgis.org/foss4g/%23foss4g.2010-06-30.log log]

* Wed June 30th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=30&hour=15&min=0&sec=0 15:00:00 UTC]
** Summary: [http://logs.qgis.org/foss4g/%23foss4g.2010-06-30.log log]

* Wed June 23rd @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=23&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on proposed layer styling
*** update on available server specs
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-06-23.log log])

* Wed June 16th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=16&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on server infrastructure
*** update on OSGeo-es dataset ([[Benchmarking 2010/How to get some sample data|Sample data download instructions]])
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-06-16.log log])
*** Attendance:
**** Jeff McKenna (jmckenna)
**** Andrea Aime (aaime)
**** Iván Sánchez (IvanSanchez)
**** Michael Smith (msmith_)
**** Cédric Briançon (cedricbr)
*** Discussion:
**** "base" run will use only shapefiles, "best" run will be any format (raster/vector)
**** teams can optionally test Oracle, PostGIS, Ingres connections during the "best" run
**** layers tested will be: labelled roads (with labels that follow the lines), thematic polygon map, and a test with labelled roads + thematic polygon + point layer in a single request (3 separate runs)
***** msmith and jmckenna to examine styling for one sheet (Barcelona, sheet# 420)
**** each base and best run will test data in raw projection (ETRS89 + UTM28/29/30/31) and then reprojected on-the-fly to the Google Mercator projection
**** threads (number of concurrent requests) will be: 1-2-4-8-16-32-64
**** US Army Corps is proposing to provide 4 desktop machines. 2 identical for testing ( win/lin ) and 1 for jmeter and 1 for databases
***** msmith will find out availability of 2-cpu machines this week

* Wed June 2nd @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=02&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** team representative introductions
*** Establish this year's 'Rules of Engagement' ([http://wiki.osgeo.org/wiki/Benchmarking_2009#Rules_of_Engagement previous RoE])
*** Discuss dataset to be used (including how to share the large dataset, for running local tests)
*** Discuss server infrastructure
*** Discuss possible changes to the test script parameters
*** Set next meeting date
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-06-02.log log])
*** Attendance:
**** Jeff McKenna, MapServer (jmckenna)
**** Cédric Briançon - Constellation (cedricbr)
**** Michael Smith - Oracle Mapviewer and Oracle connections in MapServer (msmith_)
**** Zac Spitzer - MapGuide (zacspitzer)
**** Martin Daly - Cadcorp (mpdaly)
**** Anne-Sophie Collignon - ERDAS (ascollignon)
**** Marco Hugentobler - QGIS mapserver (marco___)
**** Andrea Aime - GeoServer (aaime)
**** Dimitri Monie - ERDAS (dmonie)
**** Frank Warmerdam - MapServer, possibly Ingres as well (FrankW)
**** Iván Sánchez - provide geodata from Spanish sources and help out Mapnik (IvanSanchez)
**** Dane Springmeyer - Mapnik (springmeyer)
**** Johann Sorel - GeotoolKit / Constellation (Eclesia)
**** Adrian Custer - Constellation (acuster)
**** Pirmin Kalberer - QGIS mapserver (pirmin_k)
**** Trevor Wekel - MapGuide (trevorw)
**** Daniel Morissette - MapServer (danmo)
*** Discussion:
**** All agreed to this year's "Rules of Engagement"
***** All parties must contribute any changes that they make to their software for this exercise, back to their community. Note that the changes don't have to be contributed before the conference, just in a reasonable period of time.
***** Comparisons will be made of the best available version of the software, be it a formal release or a development version.
***** Two tests will be run: one 'baseline' test with the data in its raw format (with spatial indexes), and another 'best effort' test where 'the sky is the limit' for what changes you want to make to the data (change format, generalize, etc)
***** Teams must document all steps they did to manipulate the data/server for both the 'baseline' and 'best effort' tests. If a team does not document the steps on this wiki then that team's test results will not be used.
***** Data formats to be used will be shapefiles for vectors, and uncompressed geotiffs for rasters.
***** WMS output formats to be used will be png8 and png24 where possible
**** Dataset
***** Iván Sánchez Ortega from OSGeo-es is willing to provide local Spain datasets
***** data is for non-commercial use only
***** 10 GB of shapefiles
***** 100 GB of uncompressed rasters
**** Server infrastructure
***** possible hosts are msmith (US Army Corps) or Skygone (http://www.skygoneinc.com/)
***** need two identical servers, one running Unix and another Windows, plus a client machine
***** specs could be: quad core, 8GB of memory, 1TB hard drive
***** need fast pipe to server, as large datasets will be transferred
**** Next meeting date
***** all agreed to use same day and time next week, but limit the meeting length to one hour

== Sample Dataset Styling ==

* download [http://download.osgeo.org/benchmarking/barcelona-sample-dataset.zip sample dataset]
* layer [http://wiki.osgeo.org/wiki/Benchmarking_2010/How_to_get_some_sample_data#Vector_data_issues styling rules]
{| border="1"
|-
| '''Mapping Server'''
| '''GetMap'''
| '''SLD'''
|-
| MapServer
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 all layers]
| --
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=settlement&SRS=EPSG:25831&BBOX=426633,4579926,429740,4582528&FORMAT=image/png&WIDTH=950&HEIGHT=600 settlement]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/settlement-sld.xml settlement-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=building&SRS=EPSG:25831&BBOX=426633,4579926,429740,4582528&FORMAT=image/png&WIDTH=950&HEIGHT=600 building]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/building-sld.xml building-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=industry&SRS=EPSG:25831&BBOX=424932,4575961,427328,4577812&FORMAT=image/png&WIDTH=950&HEIGHT=600 industry]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/industry-sld.xml industry-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=contour_lines&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 contour lines]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/contour-sld.xml contour-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=motorway&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 motorway]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/motorway-sld.xml motorway-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=road&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 road]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/road-sld.xml road-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=ramp&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 ramp]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/ramp-sld.xml ramp-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=track&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 track]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/track-sld.xml track-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=point_geometry&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 point labels with geometry]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/point-geom-sld.xml point-geom-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=point_nogeom&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 point labels without geometry]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/point-nogeom-sld.xml point-nogeom-sld.xml]
|}

=== Scale dependent rules checking requests ===

These requests show the behavior of scale dependent styling around the critical scales.
The geographic area requested is always the same:

<tt>BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102</tt>

The image size changes instead to alter the scale:

* roughly 1:29k: <tt>WIDTH=824&HEIGHT=472</tt>
* roughly 1:31k: <tt>WIDTH=771&HEIGHT=442</tt>
* roughly 1:49k: <tt>WIDTH=488&HEIGHT=280</tt>
* roughly 1.51k: <tt>WIDTH=469&HEIGHT=269</tt>

The table allows to compare the various servers output. The GeoServer links use a vendor option to get the actual scale, as computed by the server, displayed at the bottom/right of the map.

{| border="1"
|-
| '''Server'''
| '''1:29k'''
| '''1:31k'''
| '''1:49k'''
| '''1:51k'''
|-
| Constellation-SDI
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
| GeoServer
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472&format_options=layout:scale 1:29k]
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442&format_options=layout:scale 1:31k]
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280&format_options=layout:scale 1:49k]
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269&format_options=layout:scale 1:51k]
|-
| Mapnik
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
| MapServer
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
| MapViewer
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
|-
| GeognoSIS
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
|-
| QGIS mapserver
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
|-
| ERDAS APOLLO
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472&srs=EPSG:4326&FORMAT=image/png 1:29k]
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442&srs=EPSG:4326&FORMAT=image/png 1:31k]
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280&height=600&srs=EPSG:4326&FORMAT=image/png 1:49k]
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269&srs=EPSG:4326&FORMAT=image/png 1:51k]
|}

== Benchmarking setup ==

Tests are going to be run using JMeter, using a progression of 1, 2, 4, 8, 16, 32, and 64 threads, each thread group doing 100, 200, 200, 400, 400, 800 requests respectively, for a total of 2200 requests.
The requests bounds and sizes are going to be picked up from a csv file generated by a random generator script.

The scripts to be run for the baseline tests, and repeated for the best effort one, are:
* raster data, JPEG output, EPSG:25831, scales between 1:1M and 1:10k
* raster data, JPEG output, EPSG:3857, scales between 1:1M and 1:10k
* vector data, PNG24 output, EPSG:4326, scales between 1:300k and 1:1k
* vector data, PNG24 output, EPSG:3875, scales between 1:300k and 1:1k
* raster + vector, JPEG output, EPSG:25831, scales between 1:300k and 1:1k
* raster + vector, JPEG output, EPSG:3785, scales between 1:300k and 1:1k

The areas to be tested are:
* 6 E, 38 N, -1 E, 43 N for vector tests
* 372360,4557880,484490,4631460 for raster tests

The raster layer in the baseline test is going to be the mosaic of GeoTIFF files Frank prepared out of the ECW files on the Linux server.

The vector data set is composed of a stack of individual layers to be stacked in the following order (bottom to top):
* settlements
* contour-0
* contour-1
* contour-2
* contour-3
* contour-4
* contour-5
* contour-6
* contour-7
* building
* industry
* track
* ramp
* road
* motorway
* point-labels-for-geometry
* point-labels-no-geometry

The best effort will use its own custom set of layers (e.g., it's likely contours will be merged into one, and so on)

== Live Benchmark WMS GetMap Requests ==

Note: sample bbox's fixed on Aug 12 to fit the proper aspect ratio of an 800/600 image request (springmeyer). Previous bbox for barcelona request was '1.8,41,2.4,42', new one used below is: '1.43333333333,41.0,2.76666666667,42.0'. Previous request for focused view was '2.1076723642349,41.407828508849,2.1178733654021,41.414271246429', new one is '2.10767236423,41.4072245022,2.1178733654,41.4148752531'. This was done to sidestep a deficiency in mapnik for affine transformation of geometries according to the WMS spec when the image w/h aspect does not match the bbox aspect, but was agreed upon as useful for these demo links by jeff and andrea in IRC.

=== MapServer ===

* Windows 64bit Server
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (zoomed-in)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (mid-zoom)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=316295,4519985.6640625,540555,4669784.3359375&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (whole-area)

* Linux
** [http://12.189.158.78:8081/cgi-bin/mapserv?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.78:8081/cgi-bin/mapserv?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev-label-fix] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev-label-fix] (labelled contours, larger view)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (zoomed-in)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (mid-zoom)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=316295,4519985.6640625,540555,4669784.3359375&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (whole-area)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle-raster-only.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg Oracle GeoRaster, MapServer 5.7-dev] (zoomed-in)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle-raster-only.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg Oracle GeoRaster, MapServer 5.7-dev] (mid-zoom)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle-raster-only.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=316295,4519985.6640625,540555,4669784.3359375&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg Oracle GeoRaster, MapServer 5.7-dev] (whole-area)

=== GeoServer ===

* [http://12.189.158.78:8080/geoserver/wms?SRS=EPSG%3A4326&LAYERS=shp_alllayers_label&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles GeoServer 2.1.x] (Barcelona extents)
*[http://12.189.158.78:8080/geoserver/wms?SRS=EPSG%3A4326&LAYERS=shp_alllayers_label&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Shapefiles GeoServer 2.1.x] (with labelled contours)
* [http://12.189.158.78:8080/geoserver/wms?EXCEPTIONS=application%2Fvnd.ogc.se_inimage&LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=2.10767236423,41.4072245022,2.1178733654,41.41487525316&WIDTH=800&HEIGHT=600 Shapefiles GeoServer 2.1.x] (labelled contours, larger view)
* [http://12.189.158.78:8080/geoserver/wms?LAYERS=ras%3Amosaic_tif&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 TIFF mosaic, zoomed in, GeoServer 2.1.x]
* [http://12.189.158.78:8080/geoserver/wms?LAYERS=ras%3Amosaic_tif&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 TIFF mosaic, mid zoom, GeoServer 2.1.x]
* [http://12.189.158.78:8080/geoserver/wms?LAYERS=ras%3Amosaic_tif&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 TIFF mosaic, whole area, GeoServer 2.1.x]

=== Mapnik ===
* [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles Mapnik2 / paleoserver] (Barcelona extents)
* [http://12.189.158.78:8091/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles Mapnik0.7.2 / mod_mapnik_wms] (Barcelona extents)
* [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Shapefiles Mapnik2 / paleoserver] (larger view - labeled contours)

=== Constellation-SDI ===
* [http://12.189.158.78:8280/constellation/WS/wms?SRS=EPSG%3A4326&LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles Constellation] (Barcelona extents)
* [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=2.10767236423,41.4072245022,2.1178733654,41.41487525316&WIDTH=800&HEIGHT=600 Shapefiles Constellation] (labelled contours, larger view)
* [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=bench:orthophoto&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 GeoTIFF Constellation] (zoomed-in)
* [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=bench:orthophoto&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600 GeoTIFF Constellation] (mid-zoom)
* [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=bench:orthophoto&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&width=800&height=600 GeoTIFF Constellation] (whole-area)

=== Oracle MapViewer ===
*[http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600&srs=EPSG:4326&FORMAT=image/png&aa=TRUE Oracle DB / MapViewer] (Barcelona Extents)

*[http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png&aa=TRUE Oracle DB / MapViewer] (with labeled contours)

=== Cadcorp GeognoSIS ===
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600 Shapefiles, GeognoSIS 7.0] (Barcelona Extents)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Shapefiles, GeognoSIS 7.0] (with labelled contours)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-postgis&BGCOLOR=0xFFFFFF&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600 PostGIS, GeognoSIS 7.0] (Barcelona Extents)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-postgis&BGCOLOR=0xFFFFFF&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 PostGIS, GeognoSIS 7.0] (with labelled contours)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-oracle&BGCOLOR=0xFFFFFF&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600 Oracle, GeognoSIS 7.0] (Barcelona Extents)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-oracle&BGCOLOR=0xFFFFFF&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Oracle, GeognoSIS 7.0] ('''no contours''' - under investigation)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-tif&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 TIFF mosaic, zoomed in, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-tif&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 TIFF mosaic, mid zoom, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-tif&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 TIFF mosaic, whole area, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-ecw&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 ECW, zoomed in, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-ecw&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 ECW, mid zoom, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-ecw&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 ECW, whole area, GeognoSIS 7.0]

=== QGIS mapserver ===
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SRS=EPSG:4326&LAYERS=shp_vector&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles, QGIS mapserver] (Barcelona Extents)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, QGIS mapserver] (with labeled contours)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/pg.qgs&SRS=EPSG:4326&LAYERS=shp_vector&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 PostGIS, QGIS mapserver] (Barcelona Extents)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/pg.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png PostGIS, QGIS mapserver] (with labeled contours)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector_raster.qgs&LAYERS=vrt&STYLES=&FORMAT=image/jpg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3043&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 TIFF mosaic, zoomed in, QGIS mapserver]
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector_raster.qgs&LAYERS=vrt&STYLES=&FORMAT=image/jpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3043&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 TIFF mosaic, mid zoom, QGIS mapserver]
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector_raster.qgs&LAYERS=vrt&STYLES=&FORMAT=image/jpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3043&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 TIFF mosaic, whole area, QGIS mapserver]

=== ERDAS APOLLO ===

Windows 64bit Server
* [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, APOLLO 10.1.2] (Barcelona extents)
* [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, APOLLO 10.1.2] (with labelled contours)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=TIFF&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 Geotiff, APOLLO 10.1.2] (Zoomed-in)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=TIFF&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 Geotiff, APOLLO 10.1.2](mid-zoom)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=TIFF&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 Geotiff, APOLLO 10.1.2] (Whole Area)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=BENCHMARK&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 ECW, APOLLO 10.1.2] (Zoomed-in)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=BENCHMARK&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 ECW, APOLLO 10.1.2] (mid-zoom)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=BENCHMARK&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 ECW, APOLLO 10.1.2] (Whole Area)

== Starting and stopping the various servers ==

* GeoServer on the Linux box:
** <tt>cd /benchmarking/geoserver</tt>
** starting: <tt>/benchmarking/geoserver/start_geoserver.sh</tt>
** stopping gracefully: <tt>/benchmarking/geoserver/stop_geoserver.sh</tt>
** check it's really dead: <tt>ps aux | grep GEOSERVER</tt> . If it hasn't stopped gracefully use <tt>kill -9 <pid></tt> to terminate the process

* MapServer and QGIS Mapserver on the Linux box (both are running in the same Apache instance):
** starting: /opt/mapserver/bin/apachectl start
** stopping: /opt/mapserver/bin/apachectl stop

* MapServer on Windows box:
** Command prompt (right-click "Run as Administrator")
** cd \ms4w
** apache-stop.bat
** apache-start.bat

* Mapnik:
** paleoserver (standalone daemon):
*** stopping: /opt/mapnik/paleoserver_stop.sh
*** starting: /opt/mapnik/paleoserver_start.sh
** mod_mapnik_wms (runs within apache):
** starting: /opt/mapnik/mod_mapnik_start.sh
** stopping: /opt/mapnik/mod_mapnik_stop.sh

* Mapviewer:
** <tt>cd /benchmarking/mapviewer</tt>
** starting: <tt> /opt/mapviewer/start_mv.sh </tt>
** stopping: <tt> /opt/mapviewer/stop_mv.sh </tt>
** check it's really dead: <tt>ps aux | grep java | grep mapviewer</tt> . If it hasn't stopped gracefully use <tt>kill -9 <pid></tt> to terminate the process

* Cadcorp GeognoSIS
** 7.0
*** Command line (with "Run as Administrator")
**** starting: net start geognosis-70
**** stopping: net stop geognosis-70
*** Or use services.msc, service name geognosis-70
** 7.1
*** Same as 7.0, but with "-71" suffix

* Erdas Windows Server
** APOLLO SDI (vector services)
*** Command Line (with "Run as Administrator")
**** starting : cd C:\opt\ERDAS\APOLLOSDI\tomcat\bin\startup.bat
**** stopping : cd C:\opt\ERDAS\APOLLOSDI\tomcat\bin\shutdown.bat
*** check the java process for erdas user has been killed
** APOLLO IWS (raster services)
*** Open IIS (Control Panel\Administrative Tools --> Internet Information Services (IIS) Manager)
**** starting : select under "Site" "Default Web Site" --> Manage Web Site : Start
**** Stopping : select under "Site" "Default Web Site" --> Manage Web Site : Stop

* Constellation on the Linux box (you must be logged in as constellation user, do not launch it as root !):
** starting: <tt>/opt/constellation/apache-tomcat-6.0.28/bin/startup.sh</tt>
** stopping: <tt>/opt/constellation/apache-tomcat-6.0.28/bin/shutdown.sh</tt>
** check it's really dead: <tt>ps aux | grep constellation</tt> . If it hasn't stopped gracefully use <tt>kill -9 <pid></tt> to terminate the process

* Oracle Database
** Login to database machine and sudo to user oracle
***Starting: lsntctl start; dbstart
***Stopping: lsntcrl stop: dbshut

== List of Software Enhancements ==

=== GeoServer ===

* support DBF larger than 2GB
* avoid reading the Z dimension when flat geometries are all that's needed
* generally improve shapefile reading: don't open the shp and dbf file if the spatial index says there is no data to read, bbox based feature skipping during filtering, limit number of disk reads to the bare minimum
* perform fast rectangular clipping of geometries before sending them to the renderer
* direct rendering path for raster to avoid the java2d bottlenecks
* improved .qix index file generation based on node load factor instead of tree depth, node collapsing for degenerate linear chains
* light parallelization of the renderer so that feature reading and rendering can be performed in parallel
* better metadata caching (feature type structure and the like)

=== MapServer ===
* large DBF support ([https://trac.osgeo.org/mapserver/ticket/3514 ticket])
* improving labels on curved lines ([http://trac.osgeo.org/mapserver/ticket/3523 ticket])
* handling label precision ([http://trac.osgeo.org/mapserver/ticket/2950 ticket])
* improving Oracle GeoRaster GDAL driver ([http://trac.osgeo.org/gdal/ticket/3746 ticket])

=== Constellation-SDI ===

Detailed in the [[Benchmarking_2010/Constellation-SDI|Constellation-SDI]] page.

=== Cadcorp GeognoSIS ===

Detailed in the [[Benchmarking_2010/Cadcorp_notes|Cadcorp notes]] page.

=== ERDAS APOLLO ===

For APOLLO IWS :
* Upgrade to GDAL 1.7.2
* Direct access to GDAL for Geotiff reader
* Improved the mosaicing of TIF datasets to remove seam lines
For APOLLO SDI :
* Tomcat 6 and Java 6: improved performances over Tomcat 5.5 and Java 5 Rendering :
* Clash management for contour labels
* Multipath rendering optimisation
* Low level code optimisation
* New option for shape index in memory

=== Mapnik ===
Detailed at: [[Benchmarking_2010/Mapnik_notes#Improvements]]

== List of Performance Results ==

=== Constellation-SDI ===

Detailed in the [[Benchmarking_2010/Constellation-SDI|Constellation-SDI]] page.

== External Related Links ==

*[[FOSS4G Benchmark|FOSS4G WMS Benchmark]]
*[http://sourceforge.net/projects/wmstester/ WMSTester - tool for testing not from OSGeo]

----

[[Category:FOSS4G2010]] [[Category:FOSS4G]]

Benchmarking 2010

2010-09-23T12:20:26Z

Wiki-Cedricbr: /* Constellation-SDI */

== Basic Premise ==

Following up on [[Benchmarking 2009|last year's exercise]], the performance shoot-out presentation at [[FOSS4G2010]] will test how long each Web mapping server takes to generate a map image, from a common set of spatial data, on a common platform. The data will be served by each Web mapping server through the WMS standard, which will serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to measure various aspects of those layers.

== Participants ==

{| border="1"
|-
| '''Mapping Server'''
| '''Development Team Leader'''
| '''IRC nick'''
| '''Server status'''
|-
| Cadcorp GeognoSIS
| [mailto:martin.daly--@--cadcorp.com Martin Daly]
| mpdaly
| Production release of server installed with preliminary configuration for SHP, PostGIS, TIF and ECW
|-
| Constellation-SDI
| [[User:Acuster|Adrian Custer]] [[User:Cedricbr|Cédric Briançon]]
| acuster cedricbr
| Running on port 8280. Installation done for Shapefiles and Rasters.
|-
| Erdas Apollo
| [mailto:dimitri.monie@erdas.com Dimitri Monie begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting]
| dmonie
| Install and configuration on-going on windows server, ecw WMS configuration done, shape and DB WMS configuration on going. Port 80, port 8080
|-
| GeoServer
| Andrea Aime
| aaime
| Normally running on port 8080 on the Linux server. Shapefiles and TIFF mosaic have been configured.
|-
| <s>MapGuide Open Source</s>
| <s>Zac Spitzer</s>
| <s>zacspitzer</s>
| ''inactive''
|-
| Mapnik
| [[User:Springmeyer|Dane Springmeyer]]
| springmeyer
| Paleoserver is running on port 8090(shape) / 8091(postgis) / 8092(raster - no reprojection). (mod_mapnik_wms is an alternate being tested, will run on same ports if active).
|-
| MapServer
| [[User:Jmckenna|Jeff McKenna]]
| jmckenna
| linux_wms_bm serving WMS requests for shps, on port 8081. windows wms_bm serving requests for shps, on port 8081.
|-
| Oracle MapViewer
| [[User:msmitherdc|Michael Smith]]
| msmith_
| Set up on port 8088
|-
| QGIS mapserver
| Marco Hugentobler
| marco___
| Set up on port 8081 (Linux server). Shapefiles and TIF (without reprojection) configured
|}

== Timeline ==

{| border="1"
|-
| January 1st, 2010
| begin contacting all mapping servers
|-
| June 1st, 2010
| commitment due by all interested mapping servers
|-
| June 2nd, 2010
| exercise begins (and weekly meetings)
|-
| August 1st, 2010
| final testing begins
|-
| August 23rd, 2010
| no further testing
|-
| September 6-9, 2010
| present results at FOSS4G2010
|}

== Rules of Engagement ==

# All parties must contribute any changes that they make to their software for this exercise, back to their community. Note that the changes don't have to be contributed before the conference, just in a reasonable period of time.
# Comparisons will be made of the best available version of the software, be it a formal release or a development version.
# Two tests will be run: one 'baseline' test with the data in its raw format (with spatial indexes), and another 'best effort' test where 'the sky is the limit' for what changes you want to make to the data (change format, generalize, etc)
# Teams must document all steps they did to manipulate the data/server for both the 'baseline' and 'best effort' tests. If a team does not document the steps on this wiki then that team's test results will not be used.
# Data formats to be used will be shapefiles for vectors, and uncompressed geotiffs for rasters.
# WMS output formats to be used will be png8 and png24 where possible

== Documenting Server Details and Differences ==

It is the responsibility of each team to document their setup with regard to data.

Please keep your notes in your servers directory in svn: http://svn.osgeo.org/osgeo/foss4g/benchmarking/

One particular thing that will differ is each team's use of spatial indexes on shapefiles.

Teams are '''not''' allowed to modify the .shp, .shx, .dbf or .prj files of the vector data (for the baseline benchmark). They can, however, create auxiliary files for spatial indexes and the like.

{| border="1"
| '''Mapping Server''' || '''ext''' || '''type''' || '''command to create'''
|-
| MapServer || .qix || quadtree || shptree <shpfile> ([[shptree notes]])
|-
| GeoServer || .qix || quadtree || Using the ones generated for MapServer atm
|-
| Mapnik || .index || quadtree || shapeindex <shpfile> ([[Benchmarking_2010/Mapnik_notes#shpindex_depth]])
|-
|Erdas || .rtr || RTree || RTreeBuilder basedir shapename max min ([[RTreeBuilder notes]])
|-
|Cadcorp || *.shp.idx || RTree || GUI command in desktop SIS ([[Benchmarking_2010/Cadcorp_notes]])
|-
| Constellation-SDI || .qix || quadtree || Using the ones generated by MapServer
|-
| MapViewer || .oix || RTree || GUI option in MapBuilder.jar, java map/theme/style builder
|}

* Note: in your server setup documentation be sure to record the EXACT command used to create the indexes, with appropriate options passed (e.g depth)

== Datasets ==

A set of data published by the spanish National Mapping Agency will be used. This data is free for non-commercial use, so it's perfectly OK to use it in the benchmark.

Vector data will be a topographic map, composed of several shapefiles (one shapefile per theme). Raster data will be 50cm/px aerial photography.

We still do not have the definitive datasets, due to their size and the processing time needed to put them together. You can get some sample data by reading [[Benchmarking 2010/How to get some sample data]].

== SVN ==

The project files (minus data) are stored in Subversion (http://svn.osgeo.org/osgeo/foss4g/benchmarking/).

== Hardware ==

Machine A (windows server)
* System Type: Dell PowerEdge R410
* Ship Date: 7/7/2010
* Processor: Intel® Xeon® E5630 2.53Ghz, 12M Cache,Turbo, HT, 1066MHz Max Mem
* 8GB Memory (4x2GB), 1333MHz Dual Ranked RDIMMs for 1Processor, Optimized
* 2TB 7.2K RPM SATA
* OS: Windows Server 64bit

Machine B (linux server)
* System Type: Dell PowerEdge R410
* Ship Date: 7/7/2010
* Processor: Intel® Xeon® E5630 2.53Ghz, 12M Cache,Turbo, HT, 1066MHz Max Mem
* 8GB Memory (4x2GB), 1333MHz Dual Ranked RDIMMs for 1Processor, Optimized
* 2TB 7.2K RPM SATA
* OS: Centos 5.5 x86-64

Machine C (DB Host)
* System Type: Gateway E6610D
* Processor, Intel Core2 Duo - E6750 2.66 Ghz
* 4Gb Ram
* 250Gb Hard Drive 7200 rpm
* OS: Centos 5.5 x86-64

Machine D (testing server (JMeter))
* System Type: Dell Precision Workstation 390
* Ship Date: 9/7/2006
* Processor, 6300, 1.86, 2M, Core Duo-conroe, Burn 2
* 2Gb RAM
* 160 Gb Hard drive 7200 rpm
* Service Tag: 5Q5LQB1
* OS: Centos 5.5 i386

== Communication ==

Coordination/communication is primarily via the Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking

Weekly meetings will occur through IRC chat in the #foss4g channel on irc.freenode.net

=== Next IRC meeting ===

* Wed August 18th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=08&day=18&hour=15&min=0&sec=0 15:00:00 UTC]
** Provisional Agenda:
*** baseline test styling equivalence: are we there yet?
*** starting and stopping all servers
*** slotting times for each team to run benchmarks
*** cleaning up the rendering benchmark blank tiles
*** graphic access to the JMeter machine
*** last testing date

=== Previous IRC meeting ===

* Wed August 11th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=08&day=11&hour=15&min=0&sec=0 15:00:00 UTC]
** Provisional Agenda:
*** Jmeter testing
**** what output formats will be requested?
**** what projections will be requested? (Platte-carré is not conformal)
**** what layer combinations will be requested? (All vector layers or various mixed subsets?)
**** what envelopes will be issued? (Frank's 2009 query generation script forces a hard cutoff at the largest scale)
**** what output metrics will be created?
**** how will the jmx file be designed? (need one .csv file per thread so all requests are different)
*** Status report from each team
*** Testing deadline: shall we extend it?
*** Can all server provide links to the sample requests and allow to build more? (style comparison, are we all doing the same work?)
*** Vector data
**** Changes in rendering (missing values for some attributes)
**** Gaps: what to do with the 9 missing sheets?
** Summary: [http://logs.qgis.org/foss4g/%23foss4g.2010-08-11.log log]

* Wed August 4th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=08&day=4&hour=15&min=0&sec=0 15:00:00 UTC]
** Provisional Agenda:
*** Machine status / issues
*** Data status / issues
**** Characters mis-encoded in shapefiles (e.g. catalan in labels): fixable? impact on PostGIS database?
**** Raster conversion to GeoTIFF (torrent of results?)
*** Styles status / issues
*** Jmeter testing
**** what output formats will be requested?
**** what projections will be requested? (Platte-carré is not conformal)
**** what layer combinations will be requested? (All vector layers or various mixed subset?)
**** what envelopes will be issued? (Frank's 2009 query generation script forces a hard cutoff at the largest scale)
**** what output metrics will be created?
**** how will the jmx file be designed? (need one .csv file per thread so all requests are different)
** Summary: [[Benchmarking 2010 IRC meeting logs Aug 4th]]

* Wed July 28th
(no info)

* Wed July 21st @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=07&day=21&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on data processing for all of Spain
*** update on server setup
*** update on data styling
*** checkin on server installs / library dependencies

* Wed July 14th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=07&day=14&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on data processing for all of Spain
*** update on server setup
*** data styling discussions
**** scales to use for layers
**** missing full road network
**** layer to use for labels along lines
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-07-14.log log])
*** Attendance:
**** Jeff McKenna (jmckenna)
**** Andrea Aime (aaime)
**** Iván Sánchez (IvanSanchez)
**** Michael Smith (msmith_)
**** Dimitri Monie (dmonie)
**** Dane Springmeyer (springmeyer)
*** Discussion:
**** data:
***** full dataset of Spain, vector and raster, has been processed by IvanSanchez, and he is checking it and will create a torrent for transfer, by the end of the week
***** because of shapefile size limit some layers may have to be split (contour-1.shp, contour-2.shp)
**** servers:
***** msmith has received both server machines
***** T1 router is being installed, it will be a dedicated line for this project
***** he is adding 2 TB drives to each
**** layer styling:
***** we notice that the Spain dataset we are using (BCN25) does not include street files (zoomed in road network)
***** we agreed to use contours for curved label tests instead
***** jmckenna will test the layer scales provided by Marco (QGIS mapserver)
***** aaime pointed out problem with the posted SLDs, in terms of the label property (jmckenna to look into it)
***** aaime pointed out that we should make sure that all servers compute the scales the same way
****** OGC recommends 90 DPI (GeoServer, Mapnik use the recommended 90...MapServer defaults to 72, but should be configurable)
****** all teams should look into how their engine handles this and report to group
**** other:
***** IvanSanchez reported that a Spanish custom-made WMS server may join the exercise (they access raster files only)
***** dmonie pointed out that Erdas has been quiet lately because one of the team leaders is on holiday

* Wed June 30th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=30&hour=15&min=0&sec=0 15:00:00 UTC]
** Summary: [http://logs.qgis.org/foss4g/%23foss4g.2010-06-30.log log]

* Wed June 30th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=30&hour=15&min=0&sec=0 15:00:00 UTC]
** Summary: [http://logs.qgis.org/foss4g/%23foss4g.2010-06-30.log log]

* Wed June 23rd @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=23&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on proposed layer styling
*** update on available server specs
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-06-23.log log])

* Wed June 16th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=16&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on server infrastructure
*** update on OSGeo-es dataset ([[Benchmarking 2010/How to get some sample data|Sample data download instructions]])
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-06-16.log log])
*** Attendance:
**** Jeff McKenna (jmckenna)
**** Andrea Aime (aaime)
**** Iván Sánchez (IvanSanchez)
**** Michael Smith (msmith_)
**** Cédric Briançon (cedricbr)
*** Discussion:
**** "base" run will use only shapefiles, "best" run will be any format (raster/vector)
**** teams can optionally test Oracle, PostGIS, Ingres connections during the "best" run
**** layers tested will be: labelled roads (with labels that follow the lines), thematic polygon map, and a test with labelled roads + thematic polygon + point layer in a single request (3 separate runs)
***** msmith and jmckenna to examine styling for one sheet (Barcelona, sheet# 420)
**** each base and best run will test data in raw projection (ETRS89 + UTM28/29/30/31) and then reprojected on-the-fly to the Google Mercator projection
**** threads (number of concurrent requests) will be: 1-2-4-8-16-32-64
**** US Army Corps is proposing to provide 4 desktop machines. 2 identical for testing ( win/lin ) and 1 for jmeter and 1 for databases
***** msmith will find out availability of 2-cpu machines this week

* Wed June 2nd @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=02&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** team representative introductions
*** Establish this year's 'Rules of Engagement' ([http://wiki.osgeo.org/wiki/Benchmarking_2009#Rules_of_Engagement previous RoE])
*** Discuss dataset to be used (including how to share the large dataset, for running local tests)
*** Discuss server infrastructure
*** Discuss possible changes to the test script parameters
*** Set next meeting date
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-06-02.log log])
*** Attendance:
**** Jeff McKenna, MapServer (jmckenna)
**** Cédric Briançon - Constellation (cedricbr)
**** Michael Smith - Oracle Mapviewer and Oracle connections in MapServer (msmith_)
**** Zac Spitzer - MapGuide (zacspitzer)
**** Martin Daly - Cadcorp (mpdaly)
**** Anne-Sophie Collignon - ERDAS (ascollignon)
**** Marco Hugentobler - QGIS mapserver (marco___)
**** Andrea Aime - GeoServer (aaime)
**** Dimitri Monie - ERDAS (dmonie)
**** Frank Warmerdam - MapServer, possibly Ingres as well (FrankW)
**** Iván Sánchez - provide geodata from Spanish sources and help out Mapnik (IvanSanchez)
**** Dane Springmeyer - Mapnik (springmeyer)
**** Johann Sorel - GeotoolKit / Constellation (Eclesia)
**** Adrian Custer - Constellation (acuster)
**** Pirmin Kalberer - QGIS mapserver (pirmin_k)
**** Trevor Wekel - MapGuide (trevorw)
**** Daniel Morissette - MapServer (danmo)
*** Discussion:
**** All agreed to this year's "Rules of Engagement"
***** All parties must contribute any changes that they make to their software for this exercise, back to their community. Note that the changes don't have to be contributed before the conference, just in a reasonable period of time.
***** Comparisons will be made of the best available version of the software, be it a formal release or a development version.
***** Two tests will be run: one 'baseline' test with the data in its raw format (with spatial indexes), and another 'best effort' test where 'the sky is the limit' for what changes you want to make to the data (change format, generalize, etc)
***** Teams must document all steps they did to manipulate the data/server for both the 'baseline' and 'best effort' tests. If a team does not document the steps on this wiki then that team's test results will not be used.
***** Data formats to be used will be shapefiles for vectors, and uncompressed geotiffs for rasters.
***** WMS output formats to be used will be png8 and png24 where possible
**** Dataset
***** Iván Sánchez Ortega from OSGeo-es is willing to provide local Spain datasets
***** data is for non-commercial use only
***** 10 GB of shapefiles
***** 100 GB of uncompressed rasters
**** Server infrastructure
***** possible hosts are msmith (US Army Corps) or Skygone (http://www.skygoneinc.com/)
***** need two identical servers, one running Unix and another Windows, plus a client machine
***** specs could be: quad core, 8GB of memory, 1TB hard drive
***** need fast pipe to server, as large datasets will be transferred
**** Next meeting date
***** all agreed to use same day and time next week, but limit the meeting length to one hour

== Sample Dataset Styling ==

* download [http://download.osgeo.org/benchmarking/barcelona-sample-dataset.zip sample dataset]
* layer [http://wiki.osgeo.org/wiki/Benchmarking_2010/How_to_get_some_sample_data#Vector_data_issues styling rules]
{| border="1"
|-
| '''Mapping Server'''
| '''GetMap'''
| '''SLD'''
|-
| MapServer
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 all layers]
| --
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=settlement&SRS=EPSG:25831&BBOX=426633,4579926,429740,4582528&FORMAT=image/png&WIDTH=950&HEIGHT=600 settlement]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/settlement-sld.xml settlement-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=building&SRS=EPSG:25831&BBOX=426633,4579926,429740,4582528&FORMAT=image/png&WIDTH=950&HEIGHT=600 building]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/building-sld.xml building-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=industry&SRS=EPSG:25831&BBOX=424932,4575961,427328,4577812&FORMAT=image/png&WIDTH=950&HEIGHT=600 industry]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/industry-sld.xml industry-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=contour_lines&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 contour lines]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/contour-sld.xml contour-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=motorway&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 motorway]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/motorway-sld.xml motorway-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=road&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 road]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/road-sld.xml road-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=ramp&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 ramp]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/ramp-sld.xml ramp-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=track&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 track]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/track-sld.xml track-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=point_geometry&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 point labels with geometry]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/point-geom-sld.xml point-geom-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=point_nogeom&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 point labels without geometry]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/point-nogeom-sld.xml point-nogeom-sld.xml]
|}

=== Scale dependent rules checking requests ===

These requests show the behavior of scale dependent styling around the critical scales.
The geographic area requested is always the same:

<tt>BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102</tt>

The image size changes instead to alter the scale:

* roughly 1:29k: <tt>WIDTH=824&HEIGHT=472</tt>
* roughly 1:31k: <tt>WIDTH=771&HEIGHT=442</tt>
* roughly 1:49k: <tt>WIDTH=488&HEIGHT=280</tt>
* roughly 1.51k: <tt>WIDTH=469&HEIGHT=269</tt>

The table allows to compare the various servers output. The GeoServer links use a vendor option to get the actual scale, as computed by the server, displayed at the bottom/right of the map.

{| border="1"
|-
| '''Server'''
| '''1:29k'''
| '''1:31k'''
| '''1:49k'''
| '''1:51k'''
|-
| Constellation-SDI
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
| GeoServer
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472&format_options=layout:scale 1:29k]
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442&format_options=layout:scale 1:31k]
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280&format_options=layout:scale 1:49k]
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269&format_options=layout:scale 1:51k]
|-
| Mapnik
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
| MapServer
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
| MapViewer
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
|-
| GeognoSIS
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
|-
| QGIS mapserver
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
|-
| ERDAS APOLLO
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472&srs=EPSG:4326&FORMAT=image/png 1:29k]
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442&srs=EPSG:4326&FORMAT=image/png 1:31k]
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280&height=600&srs=EPSG:4326&FORMAT=image/png 1:49k]
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269&srs=EPSG:4326&FORMAT=image/png 1:51k]
|}

== Benchmarking setup ==

Tests are going to be run using JMeter, using a progression of 1, 2, 4, 8, 16, 32, and 64 threads, each thread group doing 100, 200, 200, 400, 400, 800 requests respectively, for a total of 2200 requests.
The requests bounds and sizes are going to be picked up from a csv file generated by a random generator script.

The scripts to be run for the baseline tests, and repeated for the best effort one, are:
* raster data, JPEG output, EPSG:25831, scales between 1:1M and 1:10k
* raster data, JPEG output, EPSG:3857, scales between 1:1M and 1:10k
* vector data, PNG24 output, EPSG:4326, scales between 1:300k and 1:1k
* vector data, PNG24 output, EPSG:3875, scales between 1:300k and 1:1k
* raster + vector, JPEG output, EPSG:25831, scales between 1:300k and 1:1k
* raster + vector, JPEG output, EPSG:3785, scales between 1:300k and 1:1k

The areas to be tested are:
* 6 E, 38 N, -1 E, 43 N for vector tests
* 372360,4557880,484490,4631460 for raster tests

The raster layer in the baseline test is going to be the mosaic of GeoTIFF files Frank prepared out of the ECW files on the Linux server.

The vector data set is composed of a stack of individual layers to be stacked in the following order (bottom to top):
* settlements
* contour-0
* contour-1
* contour-2
* contour-3
* contour-4
* contour-5
* contour-6
* contour-7
* building
* industry
* track
* ramp
* road
* motorway
* point-labels-for-geometry
* point-labels-no-geometry

The best effort will use its own custom set of layers (e.g., it's likely contours will be merged into one, and so on)

== Live Benchmark WMS GetMap Requests ==

Note: sample bbox's fixed on Aug 12 to fit the proper aspect ratio of an 800/600 image request (springmeyer). Previous bbox for barcelona request was '1.8,41,2.4,42', new one used below is: '1.43333333333,41.0,2.76666666667,42.0'. Previous request for focused view was '2.1076723642349,41.407828508849,2.1178733654021,41.414271246429', new one is '2.10767236423,41.4072245022,2.1178733654,41.4148752531'. This was done to sidestep a deficiency in mapnik for affine transformation of geometries according to the WMS spec when the image w/h aspect does not match the bbox aspect, but was agreed upon as useful for these demo links by jeff and andrea in IRC.

=== MapServer ===

* Windows 64bit Server
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (zoomed-in)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (mid-zoom)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=316295,4519985.6640625,540555,4669784.3359375&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (whole-area)

* Linux
** [http://12.189.158.78:8081/cgi-bin/mapserv?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.78:8081/cgi-bin/mapserv?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev-label-fix] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev-label-fix] (labelled contours, larger view)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (zoomed-in)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (mid-zoom)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=316295,4519985.6640625,540555,4669784.3359375&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (whole-area)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle-raster-only.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg Oracle GeoRaster, MapServer 5.7-dev] (zoomed-in)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle-raster-only.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg Oracle GeoRaster, MapServer 5.7-dev] (mid-zoom)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle-raster-only.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=316295,4519985.6640625,540555,4669784.3359375&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg Oracle GeoRaster, MapServer 5.7-dev] (whole-area)

=== GeoServer ===

* [http://12.189.158.78:8080/geoserver/wms?SRS=EPSG%3A4326&LAYERS=shp_alllayers_label&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles GeoServer 2.1.x] (Barcelona extents)
*[http://12.189.158.78:8080/geoserver/wms?SRS=EPSG%3A4326&LAYERS=shp_alllayers_label&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Shapefiles GeoServer 2.1.x] (with labelled contours)
* [http://12.189.158.78:8080/geoserver/wms?EXCEPTIONS=application%2Fvnd.ogc.se_inimage&LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=2.10767236423,41.4072245022,2.1178733654,41.41487525316&WIDTH=800&HEIGHT=600 Shapefiles GeoServer 2.1.x] (labelled contours, larger view)
* [http://12.189.158.78:8080/geoserver/wms?LAYERS=ras%3Amosaic_tif&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 TIFF mosaic, zoomed in, GeoServer 2.1.x]
* [http://12.189.158.78:8080/geoserver/wms?LAYERS=ras%3Amosaic_tif&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 TIFF mosaic, mid zoom, GeoServer 2.1.x]
* [http://12.189.158.78:8080/geoserver/wms?LAYERS=ras%3Amosaic_tif&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 TIFF mosaic, whole area, GeoServer 2.1.x]

=== Mapnik ===
* [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles Mapnik2 / paleoserver] (Barcelona extents)
* [http://12.189.158.78:8091/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles Mapnik0.7.2 / mod_mapnik_wms] (Barcelona extents)
* [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Shapefiles Mapnik2 / paleoserver] (larger view - labeled contours)

=== Constellation-SDI ===
* [http://12.189.158.78:8280/constellation/WS/wms?SRS=EPSG%3A4326&LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles Constellation] (Barcelona extents)
* [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=2.10767236423,41.4072245022,2.1178733654,41.41487525316&WIDTH=800&HEIGHT=600 Shapefiles Constellation] (labelled contours, larger view)
* [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=bench:orthophoto&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 GeoTIFF Constellation] (zoomed-in)
* [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=bench:orthophoto&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600 GeoTIFF Constellation] (mid-zoom)

=== Oracle MapViewer ===
*[http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600&srs=EPSG:4326&FORMAT=image/png&aa=TRUE Oracle DB / MapViewer] (Barcelona Extents)

*[http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png&aa=TRUE Oracle DB / MapViewer] (with labeled contours)

=== Cadcorp GeognoSIS ===
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600 Shapefiles, GeognoSIS 7.0] (Barcelona Extents)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Shapefiles, GeognoSIS 7.0] (with labelled contours)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-postgis&BGCOLOR=0xFFFFFF&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600 PostGIS, GeognoSIS 7.0] (Barcelona Extents)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-postgis&BGCOLOR=0xFFFFFF&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 PostGIS, GeognoSIS 7.0] (with labelled contours)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-oracle&BGCOLOR=0xFFFFFF&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600 Oracle, GeognoSIS 7.0] (Barcelona Extents)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-oracle&BGCOLOR=0xFFFFFF&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Oracle, GeognoSIS 7.0] ('''no contours''' - under investigation)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-tif&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 TIFF mosaic, zoomed in, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-tif&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 TIFF mosaic, mid zoom, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-tif&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 TIFF mosaic, whole area, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-ecw&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 ECW, zoomed in, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-ecw&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 ECW, mid zoom, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-ecw&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 ECW, whole area, GeognoSIS 7.0]

=== QGIS mapserver ===
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SRS=EPSG:4326&LAYERS=shp_vector&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles, QGIS mapserver] (Barcelona Extents)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, QGIS mapserver] (with labeled contours)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/pg.qgs&SRS=EPSG:4326&LAYERS=shp_vector&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 PostGIS, QGIS mapserver] (Barcelona Extents)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/pg.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png PostGIS, QGIS mapserver] (with labeled contours)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector_raster.qgs&LAYERS=vrt&STYLES=&FORMAT=image/jpg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3043&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 TIFF mosaic, zoomed in, QGIS mapserver]
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector_raster.qgs&LAYERS=vrt&STYLES=&FORMAT=image/jpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3043&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 TIFF mosaic, mid zoom, QGIS mapserver]
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector_raster.qgs&LAYERS=vrt&STYLES=&FORMAT=image/jpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3043&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 TIFF mosaic, whole area, QGIS mapserver]

=== ERDAS APOLLO ===

Windows 64bit Server
* [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, APOLLO 10.1.2] (Barcelona extents)
* [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, APOLLO 10.1.2] (with labelled contours)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=TIFF&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 Geotiff, APOLLO 10.1.2] (Zoomed-in)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=TIFF&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 Geotiff, APOLLO 10.1.2](mid-zoom)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=TIFF&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 Geotiff, APOLLO 10.1.2] (Whole Area)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=BENCHMARK&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 ECW, APOLLO 10.1.2] (Zoomed-in)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=BENCHMARK&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 ECW, APOLLO 10.1.2] (mid-zoom)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=BENCHMARK&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 ECW, APOLLO 10.1.2] (Whole Area)

== Starting and stopping the various servers ==

* GeoServer on the Linux box:
** <tt>cd /benchmarking/geoserver</tt>
** starting: <tt>/benchmarking/geoserver/start_geoserver.sh</tt>
** stopping gracefully: <tt>/benchmarking/geoserver/stop_geoserver.sh</tt>
** check it's really dead: <tt>ps aux | grep GEOSERVER</tt> . If it hasn't stopped gracefully use <tt>kill -9 <pid></tt> to terminate the process

* MapServer and QGIS Mapserver on the Linux box (both are running in the same Apache instance):
** starting: /opt/mapserver/bin/apachectl start
** stopping: /opt/mapserver/bin/apachectl stop

* MapServer on Windows box:
** Command prompt (right-click "Run as Administrator")
** cd \ms4w
** apache-stop.bat
** apache-start.bat

* Mapnik:
** paleoserver (standalone daemon):
*** stopping: /opt/mapnik/paleoserver_stop.sh
*** starting: /opt/mapnik/paleoserver_start.sh
** mod_mapnik_wms (runs within apache):
** starting: /opt/mapnik/mod_mapnik_start.sh
** stopping: /opt/mapnik/mod_mapnik_stop.sh

* Mapviewer:
** <tt>cd /benchmarking/mapviewer</tt>
** starting: <tt> /opt/mapviewer/start_mv.sh </tt>
** stopping: <tt> /opt/mapviewer/stop_mv.sh </tt>
** check it's really dead: <tt>ps aux | grep java | grep mapviewer</tt> . If it hasn't stopped gracefully use <tt>kill -9 <pid></tt> to terminate the process

* Cadcorp GeognoSIS
** 7.0
*** Command line (with "Run as Administrator")
**** starting: net start geognosis-70
**** stopping: net stop geognosis-70
*** Or use services.msc, service name geognosis-70
** 7.1
*** Same as 7.0, but with "-71" suffix

* Erdas Windows Server
** APOLLO SDI (vector services)
*** Command Line (with "Run as Administrator")
**** starting : cd C:\opt\ERDAS\APOLLOSDI\tomcat\bin\startup.bat
**** stopping : cd C:\opt\ERDAS\APOLLOSDI\tomcat\bin\shutdown.bat
*** check the java process for erdas user has been killed
** APOLLO IWS (raster services)
*** Open IIS (Control Panel\Administrative Tools --> Internet Information Services (IIS) Manager)
**** starting : select under "Site" "Default Web Site" --> Manage Web Site : Start
**** Stopping : select under "Site" "Default Web Site" --> Manage Web Site : Stop

* Constellation on the Linux box (you must be logged in as constellation user, do not launch it as root !):
** starting: <tt>/opt/constellation/apache-tomcat-6.0.28/bin/startup.sh</tt>
** stopping: <tt>/opt/constellation/apache-tomcat-6.0.28/bin/shutdown.sh</tt>
** check it's really dead: <tt>ps aux | grep constellation</tt> . If it hasn't stopped gracefully use <tt>kill -9 <pid></tt> to terminate the process

* Oracle Database
** Login to database machine and sudo to user oracle
***Starting: lsntctl start; dbstart
***Stopping: lsntcrl stop: dbshut

== List of Software Enhancements ==

=== GeoServer ===

* support DBF larger than 2GB
* avoid reading the Z dimension when flat geometries are all that's needed
* generally improve shapefile reading: don't open the shp and dbf file if the spatial index says there is no data to read, bbox based feature skipping during filtering, limit number of disk reads to the bare minimum
* perform fast rectangular clipping of geometries before sending them to the renderer
* direct rendering path for raster to avoid the java2d bottlenecks
* improved .qix index file generation based on node load factor instead of tree depth, node collapsing for degenerate linear chains
* light parallelization of the renderer so that feature reading and rendering can be performed in parallel
* better metadata caching (feature type structure and the like)

=== MapServer ===
* large DBF support ([https://trac.osgeo.org/mapserver/ticket/3514 ticket])
* improving labels on curved lines ([http://trac.osgeo.org/mapserver/ticket/3523 ticket])
* handling label precision ([http://trac.osgeo.org/mapserver/ticket/2950 ticket])
* improving Oracle GeoRaster GDAL driver ([http://trac.osgeo.org/gdal/ticket/3746 ticket])

=== Constellation-SDI ===

Detailed in the [[Benchmarking_2010/Constellation-SDI|Constellation-SDI]] page.

=== Cadcorp GeognoSIS ===

Detailed in the [[Benchmarking_2010/Cadcorp_notes|Cadcorp notes]] page.

=== ERDAS APOLLO ===

For APOLLO IWS :
* Upgrade to GDAL 1.7.2
* Direct access to GDAL for Geotiff reader
* Improved the mosaicing of TIF datasets to remove seam lines
For APOLLO SDI :
* Tomcat 6 and Java 6: improved performances over Tomcat 5.5 and Java 5 Rendering :
* Clash management for contour labels
* Multipath rendering optimisation
* Low level code optimisation
* New option for shape index in memory

=== Mapnik ===
Detailed at: [[Benchmarking_2010/Mapnik_notes#Improvements]]

== List of Performance Results ==

=== Constellation-SDI ===

Detailed in the [[Benchmarking_2010/Constellation-SDI|Constellation-SDI]] page.

== External Related Links ==

*[[FOSS4G Benchmark|FOSS4G WMS Benchmark]]
*[http://sourceforge.net/projects/wmstester/ WMSTester - tool for testing not from OSGeo]

----

[[Category:FOSS4G2010]] [[Category:FOSS4G]]

Benchmarking 2010

2010-09-23T12:16:34Z

Wiki-Cedricbr: /* Constellation-SDI */

== Basic Premise ==

Following up on [[Benchmarking 2009|last year's exercise]], the performance shoot-out presentation at [[FOSS4G2010]] will test how long each Web mapping server takes to generate a map image, from a common set of spatial data, on a common platform. The data will be served by each Web mapping server through the WMS standard, which will serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to measure various aspects of those layers.

== Participants ==

{| border="1"
|-
| '''Mapping Server'''
| '''Development Team Leader'''
| '''IRC nick'''
| '''Server status'''
|-
| Cadcorp GeognoSIS
| [mailto:martin.daly--@--cadcorp.com Martin Daly]
| mpdaly
| Production release of server installed with preliminary configuration for SHP, PostGIS, TIF and ECW
|-
| Constellation-SDI
| [[User:Acuster|Adrian Custer]] [[User:Cedricbr|Cédric Briançon]]
| acuster cedricbr
| Running on port 8280. Installation done for Shapefiles and Rasters.
|-
| Erdas Apollo
| [mailto:dimitri.monie@erdas.com Dimitri Monie begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting]
| dmonie
| Install and configuration on-going on windows server, ecw WMS configuration done, shape and DB WMS configuration on going. Port 80, port 8080
|-
| GeoServer
| Andrea Aime
| aaime
| Normally running on port 8080 on the Linux server. Shapefiles and TIFF mosaic have been configured.
|-
| <s>MapGuide Open Source</s>
| <s>Zac Spitzer</s>
| <s>zacspitzer</s>
| ''inactive''
|-
| Mapnik
| [[User:Springmeyer|Dane Springmeyer]]
| springmeyer
| Paleoserver is running on port 8090(shape) / 8091(postgis) / 8092(raster - no reprojection). (mod_mapnik_wms is an alternate being tested, will run on same ports if active).
|-
| MapServer
| [[User:Jmckenna|Jeff McKenna]]
| jmckenna
| linux_wms_bm serving WMS requests for shps, on port 8081. windows wms_bm serving requests for shps, on port 8081.
|-
| Oracle MapViewer
| [[User:msmitherdc|Michael Smith]]
| msmith_
| Set up on port 8088
|-
| QGIS mapserver
| Marco Hugentobler
| marco___
| Set up on port 8081 (Linux server). Shapefiles and TIF (without reprojection) configured
|}

== Timeline ==

{| border="1"
|-
| January 1st, 2010
| begin contacting all mapping servers
|-
| June 1st, 2010
| commitment due by all interested mapping servers
|-
| June 2nd, 2010
| exercise begins (and weekly meetings)
|-
| August 1st, 2010
| final testing begins
|-
| August 23rd, 2010
| no further testing
|-
| September 6-9, 2010
| present results at FOSS4G2010
|}

== Rules of Engagement ==

# All parties must contribute any changes that they make to their software for this exercise, back to their community. Note that the changes don't have to be contributed before the conference, just in a reasonable period of time.
# Comparisons will be made of the best available version of the software, be it a formal release or a development version.
# Two tests will be run: one 'baseline' test with the data in its raw format (with spatial indexes), and another 'best effort' test where 'the sky is the limit' for what changes you want to make to the data (change format, generalize, etc)
# Teams must document all steps they did to manipulate the data/server for both the 'baseline' and 'best effort' tests. If a team does not document the steps on this wiki then that team's test results will not be used.
# Data formats to be used will be shapefiles for vectors, and uncompressed geotiffs for rasters.
# WMS output formats to be used will be png8 and png24 where possible

== Documenting Server Details and Differences ==

It is the responsibility of each team to document their setup with regard to data.

Please keep your notes in your servers directory in svn: http://svn.osgeo.org/osgeo/foss4g/benchmarking/

One particular thing that will differ is each team's use of spatial indexes on shapefiles.

Teams are '''not''' allowed to modify the .shp, .shx, .dbf or .prj files of the vector data (for the baseline benchmark). They can, however, create auxiliary files for spatial indexes and the like.

{| border="1"
| '''Mapping Server''' || '''ext''' || '''type''' || '''command to create'''
|-
| MapServer || .qix || quadtree || shptree <shpfile> ([[shptree notes]])
|-
| GeoServer || .qix || quadtree || Using the ones generated for MapServer atm
|-
| Mapnik || .index || quadtree || shapeindex <shpfile> ([[Benchmarking_2010/Mapnik_notes#shpindex_depth]])
|-
|Erdas || .rtr || RTree || RTreeBuilder basedir shapename max min ([[RTreeBuilder notes]])
|-
|Cadcorp || *.shp.idx || RTree || GUI command in desktop SIS ([[Benchmarking_2010/Cadcorp_notes]])
|-
| Constellation-SDI || .qix || quadtree || Using the ones generated by MapServer
|-
| MapViewer || .oix || RTree || GUI option in MapBuilder.jar, java map/theme/style builder
|}

* Note: in your server setup documentation be sure to record the EXACT command used to create the indexes, with appropriate options passed (e.g depth)

== Datasets ==

A set of data published by the spanish National Mapping Agency will be used. This data is free for non-commercial use, so it's perfectly OK to use it in the benchmark.

Vector data will be a topographic map, composed of several shapefiles (one shapefile per theme). Raster data will be 50cm/px aerial photography.

We still do not have the definitive datasets, due to their size and the processing time needed to put them together. You can get some sample data by reading [[Benchmarking 2010/How to get some sample data]].

== SVN ==

The project files (minus data) are stored in Subversion (http://svn.osgeo.org/osgeo/foss4g/benchmarking/).

== Hardware ==

Machine A (windows server)
* System Type: Dell PowerEdge R410
* Ship Date: 7/7/2010
* Processor: Intel® Xeon® E5630 2.53Ghz, 12M Cache,Turbo, HT, 1066MHz Max Mem
* 8GB Memory (4x2GB), 1333MHz Dual Ranked RDIMMs for 1Processor, Optimized
* 2TB 7.2K RPM SATA
* OS: Windows Server 64bit

Machine B (linux server)
* System Type: Dell PowerEdge R410
* Ship Date: 7/7/2010
* Processor: Intel® Xeon® E5630 2.53Ghz, 12M Cache,Turbo, HT, 1066MHz Max Mem
* 8GB Memory (4x2GB), 1333MHz Dual Ranked RDIMMs for 1Processor, Optimized
* 2TB 7.2K RPM SATA
* OS: Centos 5.5 x86-64

Machine C (DB Host)
* System Type: Gateway E6610D
* Processor, Intel Core2 Duo - E6750 2.66 Ghz
* 4Gb Ram
* 250Gb Hard Drive 7200 rpm
* OS: Centos 5.5 x86-64

Machine D (testing server (JMeter))
* System Type: Dell Precision Workstation 390
* Ship Date: 9/7/2006
* Processor, 6300, 1.86, 2M, Core Duo-conroe, Burn 2
* 2Gb RAM
* 160 Gb Hard drive 7200 rpm
* Service Tag: 5Q5LQB1
* OS: Centos 5.5 i386

== Communication ==

Coordination/communication is primarily via the Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking

Weekly meetings will occur through IRC chat in the #foss4g channel on irc.freenode.net

=== Next IRC meeting ===

* Wed August 18th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=08&day=18&hour=15&min=0&sec=0 15:00:00 UTC]
** Provisional Agenda:
*** baseline test styling equivalence: are we there yet?
*** starting and stopping all servers
*** slotting times for each team to run benchmarks
*** cleaning up the rendering benchmark blank tiles
*** graphic access to the JMeter machine
*** last testing date

=== Previous IRC meeting ===

* Wed August 11th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=08&day=11&hour=15&min=0&sec=0 15:00:00 UTC]
** Provisional Agenda:
*** Jmeter testing
**** what output formats will be requested?
**** what projections will be requested? (Platte-carré is not conformal)
**** what layer combinations will be requested? (All vector layers or various mixed subsets?)
**** what envelopes will be issued? (Frank's 2009 query generation script forces a hard cutoff at the largest scale)
**** what output metrics will be created?
**** how will the jmx file be designed? (need one .csv file per thread so all requests are different)
*** Status report from each team
*** Testing deadline: shall we extend it?
*** Can all server provide links to the sample requests and allow to build more? (style comparison, are we all doing the same work?)
*** Vector data
**** Changes in rendering (missing values for some attributes)
**** Gaps: what to do with the 9 missing sheets?
** Summary: [http://logs.qgis.org/foss4g/%23foss4g.2010-08-11.log log]

* Wed August 4th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=08&day=4&hour=15&min=0&sec=0 15:00:00 UTC]
** Provisional Agenda:
*** Machine status / issues
*** Data status / issues
**** Characters mis-encoded in shapefiles (e.g. catalan in labels): fixable? impact on PostGIS database?
**** Raster conversion to GeoTIFF (torrent of results?)
*** Styles status / issues
*** Jmeter testing
**** what output formats will be requested?
**** what projections will be requested? (Platte-carré is not conformal)
**** what layer combinations will be requested? (All vector layers or various mixed subset?)
**** what envelopes will be issued? (Frank's 2009 query generation script forces a hard cutoff at the largest scale)
**** what output metrics will be created?
**** how will the jmx file be designed? (need one .csv file per thread so all requests are different)
** Summary: [[Benchmarking 2010 IRC meeting logs Aug 4th]]

* Wed July 28th
(no info)

* Wed July 21st @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=07&day=21&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on data processing for all of Spain
*** update on server setup
*** update on data styling
*** checkin on server installs / library dependencies

* Wed July 14th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=07&day=14&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on data processing for all of Spain
*** update on server setup
*** data styling discussions
**** scales to use for layers
**** missing full road network
**** layer to use for labels along lines
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-07-14.log log])
*** Attendance:
**** Jeff McKenna (jmckenna)
**** Andrea Aime (aaime)
**** Iván Sánchez (IvanSanchez)
**** Michael Smith (msmith_)
**** Dimitri Monie (dmonie)
**** Dane Springmeyer (springmeyer)
*** Discussion:
**** data:
***** full dataset of Spain, vector and raster, has been processed by IvanSanchez, and he is checking it and will create a torrent for transfer, by the end of the week
***** because of shapefile size limit some layers may have to be split (contour-1.shp, contour-2.shp)
**** servers:
***** msmith has received both server machines
***** T1 router is being installed, it will be a dedicated line for this project
***** he is adding 2 TB drives to each
**** layer styling:
***** we notice that the Spain dataset we are using (BCN25) does not include street files (zoomed in road network)
***** we agreed to use contours for curved label tests instead
***** jmckenna will test the layer scales provided by Marco (QGIS mapserver)
***** aaime pointed out problem with the posted SLDs, in terms of the label property (jmckenna to look into it)
***** aaime pointed out that we should make sure that all servers compute the scales the same way
****** OGC recommends 90 DPI (GeoServer, Mapnik use the recommended 90...MapServer defaults to 72, but should be configurable)
****** all teams should look into how their engine handles this and report to group
**** other:
***** IvanSanchez reported that a Spanish custom-made WMS server may join the exercise (they access raster files only)
***** dmonie pointed out that Erdas has been quiet lately because one of the team leaders is on holiday

* Wed June 30th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=30&hour=15&min=0&sec=0 15:00:00 UTC]
** Summary: [http://logs.qgis.org/foss4g/%23foss4g.2010-06-30.log log]

* Wed June 30th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=30&hour=15&min=0&sec=0 15:00:00 UTC]
** Summary: [http://logs.qgis.org/foss4g/%23foss4g.2010-06-30.log log]

* Wed June 23rd @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=23&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on proposed layer styling
*** update on available server specs
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-06-23.log log])

* Wed June 16th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=16&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on server infrastructure
*** update on OSGeo-es dataset ([[Benchmarking 2010/How to get some sample data|Sample data download instructions]])
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-06-16.log log])
*** Attendance:
**** Jeff McKenna (jmckenna)
**** Andrea Aime (aaime)
**** Iván Sánchez (IvanSanchez)
**** Michael Smith (msmith_)
**** Cédric Briançon (cedricbr)
*** Discussion:
**** "base" run will use only shapefiles, "best" run will be any format (raster/vector)
**** teams can optionally test Oracle, PostGIS, Ingres connections during the "best" run
**** layers tested will be: labelled roads (with labels that follow the lines), thematic polygon map, and a test with labelled roads + thematic polygon + point layer in a single request (3 separate runs)
***** msmith and jmckenna to examine styling for one sheet (Barcelona, sheet# 420)
**** each base and best run will test data in raw projection (ETRS89 + UTM28/29/30/31) and then reprojected on-the-fly to the Google Mercator projection
**** threads (number of concurrent requests) will be: 1-2-4-8-16-32-64
**** US Army Corps is proposing to provide 4 desktop machines. 2 identical for testing ( win/lin ) and 1 for jmeter and 1 for databases
***** msmith will find out availability of 2-cpu machines this week

* Wed June 2nd @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=02&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** team representative introductions
*** Establish this year's 'Rules of Engagement' ([http://wiki.osgeo.org/wiki/Benchmarking_2009#Rules_of_Engagement previous RoE])
*** Discuss dataset to be used (including how to share the large dataset, for running local tests)
*** Discuss server infrastructure
*** Discuss possible changes to the test script parameters
*** Set next meeting date
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-06-02.log log])
*** Attendance:
**** Jeff McKenna, MapServer (jmckenna)
**** Cédric Briançon - Constellation (cedricbr)
**** Michael Smith - Oracle Mapviewer and Oracle connections in MapServer (msmith_)
**** Zac Spitzer - MapGuide (zacspitzer)
**** Martin Daly - Cadcorp (mpdaly)
**** Anne-Sophie Collignon - ERDAS (ascollignon)
**** Marco Hugentobler - QGIS mapserver (marco___)
**** Andrea Aime - GeoServer (aaime)
**** Dimitri Monie - ERDAS (dmonie)
**** Frank Warmerdam - MapServer, possibly Ingres as well (FrankW)
**** Iván Sánchez - provide geodata from Spanish sources and help out Mapnik (IvanSanchez)
**** Dane Springmeyer - Mapnik (springmeyer)
**** Johann Sorel - GeotoolKit / Constellation (Eclesia)
**** Adrian Custer - Constellation (acuster)
**** Pirmin Kalberer - QGIS mapserver (pirmin_k)
**** Trevor Wekel - MapGuide (trevorw)
**** Daniel Morissette - MapServer (danmo)
*** Discussion:
**** All agreed to this year's "Rules of Engagement"
***** All parties must contribute any changes that they make to their software for this exercise, back to their community. Note that the changes don't have to be contributed before the conference, just in a reasonable period of time.
***** Comparisons will be made of the best available version of the software, be it a formal release or a development version.
***** Two tests will be run: one 'baseline' test with the data in its raw format (with spatial indexes), and another 'best effort' test where 'the sky is the limit' for what changes you want to make to the data (change format, generalize, etc)
***** Teams must document all steps they did to manipulate the data/server for both the 'baseline' and 'best effort' tests. If a team does not document the steps on this wiki then that team's test results will not be used.
***** Data formats to be used will be shapefiles for vectors, and uncompressed geotiffs for rasters.
***** WMS output formats to be used will be png8 and png24 where possible
**** Dataset
***** Iván Sánchez Ortega from OSGeo-es is willing to provide local Spain datasets
***** data is for non-commercial use only
***** 10 GB of shapefiles
***** 100 GB of uncompressed rasters
**** Server infrastructure
***** possible hosts are msmith (US Army Corps) or Skygone (http://www.skygoneinc.com/)
***** need two identical servers, one running Unix and another Windows, plus a client machine
***** specs could be: quad core, 8GB of memory, 1TB hard drive
***** need fast pipe to server, as large datasets will be transferred
**** Next meeting date
***** all agreed to use same day and time next week, but limit the meeting length to one hour

== Sample Dataset Styling ==

* download [http://download.osgeo.org/benchmarking/barcelona-sample-dataset.zip sample dataset]
* layer [http://wiki.osgeo.org/wiki/Benchmarking_2010/How_to_get_some_sample_data#Vector_data_issues styling rules]
{| border="1"
|-
| '''Mapping Server'''
| '''GetMap'''
| '''SLD'''
|-
| MapServer
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 all layers]
| --
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=settlement&SRS=EPSG:25831&BBOX=426633,4579926,429740,4582528&FORMAT=image/png&WIDTH=950&HEIGHT=600 settlement]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/settlement-sld.xml settlement-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=building&SRS=EPSG:25831&BBOX=426633,4579926,429740,4582528&FORMAT=image/png&WIDTH=950&HEIGHT=600 building]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/building-sld.xml building-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=industry&SRS=EPSG:25831&BBOX=424932,4575961,427328,4577812&FORMAT=image/png&WIDTH=950&HEIGHT=600 industry]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/industry-sld.xml industry-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=contour_lines&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 contour lines]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/contour-sld.xml contour-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=motorway&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 motorway]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/motorway-sld.xml motorway-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=road&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 road]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/road-sld.xml road-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=ramp&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 ramp]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/ramp-sld.xml ramp-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=track&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 track]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/track-sld.xml track-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=point_geometry&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 point labels with geometry]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/point-geom-sld.xml point-geom-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=point_nogeom&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 point labels without geometry]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/point-nogeom-sld.xml point-nogeom-sld.xml]
|}

=== Scale dependent rules checking requests ===

These requests show the behavior of scale dependent styling around the critical scales.
The geographic area requested is always the same:

<tt>BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102</tt>

The image size changes instead to alter the scale:

* roughly 1:29k: <tt>WIDTH=824&HEIGHT=472</tt>
* roughly 1:31k: <tt>WIDTH=771&HEIGHT=442</tt>
* roughly 1:49k: <tt>WIDTH=488&HEIGHT=280</tt>
* roughly 1.51k: <tt>WIDTH=469&HEIGHT=269</tt>

The table allows to compare the various servers output. The GeoServer links use a vendor option to get the actual scale, as computed by the server, displayed at the bottom/right of the map.

{| border="1"
|-
| '''Server'''
| '''1:29k'''
| '''1:31k'''
| '''1:49k'''
| '''1:51k'''
|-
| Constellation-SDI
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
| GeoServer
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472&format_options=layout:scale 1:29k]
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442&format_options=layout:scale 1:31k]
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280&format_options=layout:scale 1:49k]
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269&format_options=layout:scale 1:51k]
|-
| Mapnik
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
| MapServer
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
| MapViewer
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
|-
| GeognoSIS
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
|-
| QGIS mapserver
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
|-
| ERDAS APOLLO
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472&srs=EPSG:4326&FORMAT=image/png 1:29k]
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442&srs=EPSG:4326&FORMAT=image/png 1:31k]
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280&height=600&srs=EPSG:4326&FORMAT=image/png 1:49k]
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269&srs=EPSG:4326&FORMAT=image/png 1:51k]
|}

== Benchmarking setup ==

Tests are going to be run using JMeter, using a progression of 1, 2, 4, 8, 16, 32, and 64 threads, each thread group doing 100, 200, 200, 400, 400, 800 requests respectively, for a total of 2200 requests.
The requests bounds and sizes are going to be picked up from a csv file generated by a random generator script.

The scripts to be run for the baseline tests, and repeated for the best effort one, are:
* raster data, JPEG output, EPSG:25831, scales between 1:1M and 1:10k
* raster data, JPEG output, EPSG:3857, scales between 1:1M and 1:10k
* vector data, PNG24 output, EPSG:4326, scales between 1:300k and 1:1k
* vector data, PNG24 output, EPSG:3875, scales between 1:300k and 1:1k
* raster + vector, JPEG output, EPSG:25831, scales between 1:300k and 1:1k
* raster + vector, JPEG output, EPSG:3785, scales between 1:300k and 1:1k

The areas to be tested are:
* 6 E, 38 N, -1 E, 43 N for vector tests
* 372360,4557880,484490,4631460 for raster tests

The raster layer in the baseline test is going to be the mosaic of GeoTIFF files Frank prepared out of the ECW files on the Linux server.

The vector data set is composed of a stack of individual layers to be stacked in the following order (bottom to top):
* settlements
* contour-0
* contour-1
* contour-2
* contour-3
* contour-4
* contour-5
* contour-6
* contour-7
* building
* industry
* track
* ramp
* road
* motorway
* point-labels-for-geometry
* point-labels-no-geometry

The best effort will use its own custom set of layers (e.g., it's likely contours will be merged into one, and so on)

== Live Benchmark WMS GetMap Requests ==

Note: sample bbox's fixed on Aug 12 to fit the proper aspect ratio of an 800/600 image request (springmeyer). Previous bbox for barcelona request was '1.8,41,2.4,42', new one used below is: '1.43333333333,41.0,2.76666666667,42.0'. Previous request for focused view was '2.1076723642349,41.407828508849,2.1178733654021,41.414271246429', new one is '2.10767236423,41.4072245022,2.1178733654,41.4148752531'. This was done to sidestep a deficiency in mapnik for affine transformation of geometries according to the WMS spec when the image w/h aspect does not match the bbox aspect, but was agreed upon as useful for these demo links by jeff and andrea in IRC.

=== MapServer ===

* Windows 64bit Server
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (zoomed-in)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (mid-zoom)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=316295,4519985.6640625,540555,4669784.3359375&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (whole-area)

* Linux
** [http://12.189.158.78:8081/cgi-bin/mapserv?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.78:8081/cgi-bin/mapserv?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev-label-fix] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev-label-fix] (labelled contours, larger view)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (zoomed-in)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (mid-zoom)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=316295,4519985.6640625,540555,4669784.3359375&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (whole-area)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle-raster-only.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg Oracle GeoRaster, MapServer 5.7-dev] (zoomed-in)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle-raster-only.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg Oracle GeoRaster, MapServer 5.7-dev] (mid-zoom)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle-raster-only.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=316295,4519985.6640625,540555,4669784.3359375&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg Oracle GeoRaster, MapServer 5.7-dev] (whole-area)

=== GeoServer ===

* [http://12.189.158.78:8080/geoserver/wms?SRS=EPSG%3A4326&LAYERS=shp_alllayers_label&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles GeoServer 2.1.x] (Barcelona extents)
*[http://12.189.158.78:8080/geoserver/wms?SRS=EPSG%3A4326&LAYERS=shp_alllayers_label&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Shapefiles GeoServer 2.1.x] (with labelled contours)
* [http://12.189.158.78:8080/geoserver/wms?EXCEPTIONS=application%2Fvnd.ogc.se_inimage&LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=2.10767236423,41.4072245022,2.1178733654,41.41487525316&WIDTH=800&HEIGHT=600 Shapefiles GeoServer 2.1.x] (labelled contours, larger view)
* [http://12.189.158.78:8080/geoserver/wms?LAYERS=ras%3Amosaic_tif&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 TIFF mosaic, zoomed in, GeoServer 2.1.x]
* [http://12.189.158.78:8080/geoserver/wms?LAYERS=ras%3Amosaic_tif&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 TIFF mosaic, mid zoom, GeoServer 2.1.x]
* [http://12.189.158.78:8080/geoserver/wms?LAYERS=ras%3Amosaic_tif&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 TIFF mosaic, whole area, GeoServer 2.1.x]

=== Mapnik ===
* [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles Mapnik2 / paleoserver] (Barcelona extents)
* [http://12.189.158.78:8091/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles Mapnik0.7.2 / mod_mapnik_wms] (Barcelona extents)
* [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Shapefiles Mapnik2 / paleoserver] (larger view - labeled contours)

=== Constellation-SDI ===
* [http://12.189.158.78:8280/constellation/WS/wms?SRS=EPSG%3A4326&LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles Constellation] (Barcelona extents)
* [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=2.10767236423,41.4072245022,2.1178733654,41.41487525316&WIDTH=800&HEIGHT=600 Shapefiles Constellation] (labelled contours, larger view)
* [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=bench:orthophoto&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 GeoTIFF Constellation] (zoomed-in)

=== Oracle MapViewer ===
*[http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600&srs=EPSG:4326&FORMAT=image/png&aa=TRUE Oracle DB / MapViewer] (Barcelona Extents)

*[http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png&aa=TRUE Oracle DB / MapViewer] (with labeled contours)

=== Cadcorp GeognoSIS ===
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600 Shapefiles, GeognoSIS 7.0] (Barcelona Extents)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Shapefiles, GeognoSIS 7.0] (with labelled contours)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-postgis&BGCOLOR=0xFFFFFF&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600 PostGIS, GeognoSIS 7.0] (Barcelona Extents)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-postgis&BGCOLOR=0xFFFFFF&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 PostGIS, GeognoSIS 7.0] (with labelled contours)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-oracle&BGCOLOR=0xFFFFFF&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600 Oracle, GeognoSIS 7.0] (Barcelona Extents)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-oracle&BGCOLOR=0xFFFFFF&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Oracle, GeognoSIS 7.0] ('''no contours''' - under investigation)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-tif&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 TIFF mosaic, zoomed in, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-tif&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 TIFF mosaic, mid zoom, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-tif&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 TIFF mosaic, whole area, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-ecw&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 ECW, zoomed in, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-ecw&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 ECW, mid zoom, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-ecw&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 ECW, whole area, GeognoSIS 7.0]

=== QGIS mapserver ===
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SRS=EPSG:4326&LAYERS=shp_vector&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles, QGIS mapserver] (Barcelona Extents)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, QGIS mapserver] (with labeled contours)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/pg.qgs&SRS=EPSG:4326&LAYERS=shp_vector&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 PostGIS, QGIS mapserver] (Barcelona Extents)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/pg.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png PostGIS, QGIS mapserver] (with labeled contours)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector_raster.qgs&LAYERS=vrt&STYLES=&FORMAT=image/jpg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3043&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 TIFF mosaic, zoomed in, QGIS mapserver]
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector_raster.qgs&LAYERS=vrt&STYLES=&FORMAT=image/jpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3043&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 TIFF mosaic, mid zoom, QGIS mapserver]
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector_raster.qgs&LAYERS=vrt&STYLES=&FORMAT=image/jpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3043&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 TIFF mosaic, whole area, QGIS mapserver]

=== ERDAS APOLLO ===

Windows 64bit Server
* [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, APOLLO 10.1.2] (Barcelona extents)
* [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, APOLLO 10.1.2] (with labelled contours)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=TIFF&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 Geotiff, APOLLO 10.1.2] (Zoomed-in)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=TIFF&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 Geotiff, APOLLO 10.1.2](mid-zoom)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=TIFF&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 Geotiff, APOLLO 10.1.2] (Whole Area)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=BENCHMARK&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 ECW, APOLLO 10.1.2] (Zoomed-in)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=BENCHMARK&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 ECW, APOLLO 10.1.2] (mid-zoom)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=BENCHMARK&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 ECW, APOLLO 10.1.2] (Whole Area)

== Starting and stopping the various servers ==

* GeoServer on the Linux box:
** <tt>cd /benchmarking/geoserver</tt>
** starting: <tt>/benchmarking/geoserver/start_geoserver.sh</tt>
** stopping gracefully: <tt>/benchmarking/geoserver/stop_geoserver.sh</tt>
** check it's really dead: <tt>ps aux | grep GEOSERVER</tt> . If it hasn't stopped gracefully use <tt>kill -9 <pid></tt> to terminate the process

* MapServer and QGIS Mapserver on the Linux box (both are running in the same Apache instance):
** starting: /opt/mapserver/bin/apachectl start
** stopping: /opt/mapserver/bin/apachectl stop

* MapServer on Windows box:
** Command prompt (right-click "Run as Administrator")
** cd \ms4w
** apache-stop.bat
** apache-start.bat

* Mapnik:
** paleoserver (standalone daemon):
*** stopping: /opt/mapnik/paleoserver_stop.sh
*** starting: /opt/mapnik/paleoserver_start.sh
** mod_mapnik_wms (runs within apache):
** starting: /opt/mapnik/mod_mapnik_start.sh
** stopping: /opt/mapnik/mod_mapnik_stop.sh

* Mapviewer:
** <tt>cd /benchmarking/mapviewer</tt>
** starting: <tt> /opt/mapviewer/start_mv.sh </tt>
** stopping: <tt> /opt/mapviewer/stop_mv.sh </tt>
** check it's really dead: <tt>ps aux | grep java | grep mapviewer</tt> . If it hasn't stopped gracefully use <tt>kill -9 <pid></tt> to terminate the process

* Cadcorp GeognoSIS
** 7.0
*** Command line (with "Run as Administrator")
**** starting: net start geognosis-70
**** stopping: net stop geognosis-70
*** Or use services.msc, service name geognosis-70
** 7.1
*** Same as 7.0, but with "-71" suffix

* Erdas Windows Server
** APOLLO SDI (vector services)
*** Command Line (with "Run as Administrator")
**** starting : cd C:\opt\ERDAS\APOLLOSDI\tomcat\bin\startup.bat
**** stopping : cd C:\opt\ERDAS\APOLLOSDI\tomcat\bin\shutdown.bat
*** check the java process for erdas user has been killed
** APOLLO IWS (raster services)
*** Open IIS (Control Panel\Administrative Tools --> Internet Information Services (IIS) Manager)
**** starting : select under "Site" "Default Web Site" --> Manage Web Site : Start
**** Stopping : select under "Site" "Default Web Site" --> Manage Web Site : Stop

* Constellation on the Linux box (you must be logged in as constellation user, do not launch it as root !):
** starting: <tt>/opt/constellation/apache-tomcat-6.0.28/bin/startup.sh</tt>
** stopping: <tt>/opt/constellation/apache-tomcat-6.0.28/bin/shutdown.sh</tt>
** check it's really dead: <tt>ps aux | grep constellation</tt> . If it hasn't stopped gracefully use <tt>kill -9 <pid></tt> to terminate the process

* Oracle Database
** Login to database machine and sudo to user oracle
***Starting: lsntctl start; dbstart
***Stopping: lsntcrl stop: dbshut

== List of Software Enhancements ==

=== GeoServer ===

* support DBF larger than 2GB
* avoid reading the Z dimension when flat geometries are all that's needed
* generally improve shapefile reading: don't open the shp and dbf file if the spatial index says there is no data to read, bbox based feature skipping during filtering, limit number of disk reads to the bare minimum
* perform fast rectangular clipping of geometries before sending them to the renderer
* direct rendering path for raster to avoid the java2d bottlenecks
* improved .qix index file generation based on node load factor instead of tree depth, node collapsing for degenerate linear chains
* light parallelization of the renderer so that feature reading and rendering can be performed in parallel
* better metadata caching (feature type structure and the like)

=== MapServer ===
* large DBF support ([https://trac.osgeo.org/mapserver/ticket/3514 ticket])
* improving labels on curved lines ([http://trac.osgeo.org/mapserver/ticket/3523 ticket])
* handling label precision ([http://trac.osgeo.org/mapserver/ticket/2950 ticket])
* improving Oracle GeoRaster GDAL driver ([http://trac.osgeo.org/gdal/ticket/3746 ticket])

=== Constellation-SDI ===

Detailed in the [[Benchmarking_2010/Constellation-SDI|Constellation-SDI]] page.

=== Cadcorp GeognoSIS ===

Detailed in the [[Benchmarking_2010/Cadcorp_notes|Cadcorp notes]] page.

=== ERDAS APOLLO ===

For APOLLO IWS :
* Upgrade to GDAL 1.7.2
* Direct access to GDAL for Geotiff reader
* Improved the mosaicing of TIF datasets to remove seam lines
For APOLLO SDI :
* Tomcat 6 and Java 6: improved performances over Tomcat 5.5 and Java 5 Rendering :
* Clash management for contour labels
* Multipath rendering optimisation
* Low level code optimisation
* New option for shape index in memory

=== Mapnik ===
Detailed at: [[Benchmarking_2010/Mapnik_notes#Improvements]]

== List of Performance Results ==

=== Constellation-SDI ===

Detailed in the [[Benchmarking_2010/Constellation-SDI|Constellation-SDI]] page.

== External Related Links ==

*[[FOSS4G Benchmark|FOSS4G WMS Benchmark]]
*[http://sourceforge.net/projects/wmstester/ WMSTester - tool for testing not from OSGeo]

----

[[Category:FOSS4G2010]] [[Category:FOSS4G]]

Benchmarking 2010

2010-09-23T09:58:57Z

Wiki-Cedricbr: /* Starting and stopping the various servers */

== Basic Premise ==

Following up on [[Benchmarking 2009|last year's exercise]], the performance shoot-out presentation at [[FOSS4G2010]] will test how long each Web mapping server takes to generate a map image, from a common set of spatial data, on a common platform. The data will be served by each Web mapping server through the WMS standard, which will serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to measure various aspects of those layers.

== Participants ==

{| border="1"
|-
| '''Mapping Server'''
| '''Development Team Leader'''
| '''IRC nick'''
| '''Server status'''
|-
| Cadcorp GeognoSIS
| [mailto:martin.daly--@--cadcorp.com Martin Daly]
| mpdaly
| Production release of server installed with preliminary configuration for SHP, PostGIS, TIF and ECW
|-
| Constellation-SDI
| [[User:Acuster|Adrian Custer]] [[User:Cedricbr|Cédric Briançon]]
| acuster cedricbr
| Running on port 8280. Installation done for Shapefiles and Rasters.
|-
| Erdas Apollo
| [mailto:dimitri.monie@erdas.com Dimitri Monie begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting]
| dmonie
| Install and configuration on-going on windows server, ecw WMS configuration done, shape and DB WMS configuration on going. Port 80, port 8080
|-
| GeoServer
| Andrea Aime
| aaime
| Normally running on port 8080 on the Linux server. Shapefiles and TIFF mosaic have been configured.
|-
| <s>MapGuide Open Source</s>
| <s>Zac Spitzer</s>
| <s>zacspitzer</s>
| ''inactive''
|-
| Mapnik
| [[User:Springmeyer|Dane Springmeyer]]
| springmeyer
| Paleoserver is running on port 8090(shape) / 8091(postgis) / 8092(raster - no reprojection). (mod_mapnik_wms is an alternate being tested, will run on same ports if active).
|-
| MapServer
| [[User:Jmckenna|Jeff McKenna]]
| jmckenna
| linux_wms_bm serving WMS requests for shps, on port 8081. windows wms_bm serving requests for shps, on port 8081.
|-
| Oracle MapViewer
| [[User:msmitherdc|Michael Smith]]
| msmith_
| Set up on port 8088
|-
| QGIS mapserver
| Marco Hugentobler
| marco___
| Set up on port 8081 (Linux server). Shapefiles and TIF (without reprojection) configured
|}

== Timeline ==

{| border="1"
|-
| January 1st, 2010
| begin contacting all mapping servers
|-
| June 1st, 2010
| commitment due by all interested mapping servers
|-
| June 2nd, 2010
| exercise begins (and weekly meetings)
|-
| August 1st, 2010
| final testing begins
|-
| August 23rd, 2010
| no further testing
|-
| September 6-9, 2010
| present results at FOSS4G2010
|}

== Rules of Engagement ==

# All parties must contribute any changes that they make to their software for this exercise, back to their community. Note that the changes don't have to be contributed before the conference, just in a reasonable period of time.
# Comparisons will be made of the best available version of the software, be it a formal release or a development version.
# Two tests will be run: one 'baseline' test with the data in its raw format (with spatial indexes), and another 'best effort' test where 'the sky is the limit' for what changes you want to make to the data (change format, generalize, etc)
# Teams must document all steps they did to manipulate the data/server for both the 'baseline' and 'best effort' tests. If a team does not document the steps on this wiki then that team's test results will not be used.
# Data formats to be used will be shapefiles for vectors, and uncompressed geotiffs for rasters.
# WMS output formats to be used will be png8 and png24 where possible

== Documenting Server Details and Differences ==

It is the responsibility of each team to document their setup with regard to data.

Please keep your notes in your servers directory in svn: http://svn.osgeo.org/osgeo/foss4g/benchmarking/

One particular thing that will differ is each team's use of spatial indexes on shapefiles.

Teams are '''not''' allowed to modify the .shp, .shx, .dbf or .prj files of the vector data (for the baseline benchmark). They can, however, create auxiliary files for spatial indexes and the like.

{| border="1"
| '''Mapping Server''' || '''ext''' || '''type''' || '''command to create'''
|-
| MapServer || .qix || quadtree || shptree <shpfile> ([[shptree notes]])
|-
| GeoServer || .qix || quadtree || Using the ones generated for MapServer atm
|-
| Mapnik || .index || quadtree || shapeindex <shpfile> ([[Benchmarking_2010/Mapnik_notes#shpindex_depth]])
|-
|Erdas || .rtr || RTree || RTreeBuilder basedir shapename max min ([[RTreeBuilder notes]])
|-
|Cadcorp || *.shp.idx || RTree || GUI command in desktop SIS ([[Benchmarking_2010/Cadcorp_notes]])
|-
| Constellation-SDI || .qix || quadtree || Using the ones generated by MapServer
|-
| MapViewer || .oix || RTree || GUI option in MapBuilder.jar, java map/theme/style builder
|}

* Note: in your server setup documentation be sure to record the EXACT command used to create the indexes, with appropriate options passed (e.g depth)

== Datasets ==

A set of data published by the spanish National Mapping Agency will be used. This data is free for non-commercial use, so it's perfectly OK to use it in the benchmark.

Vector data will be a topographic map, composed of several shapefiles (one shapefile per theme). Raster data will be 50cm/px aerial photography.

We still do not have the definitive datasets, due to their size and the processing time needed to put them together. You can get some sample data by reading [[Benchmarking 2010/How to get some sample data]].

== SVN ==

The project files (minus data) are stored in Subversion (http://svn.osgeo.org/osgeo/foss4g/benchmarking/).

== Hardware ==

Machine A (windows server)
* System Type: Dell PowerEdge R410
* Ship Date: 7/7/2010
* Processor: Intel® Xeon® E5630 2.53Ghz, 12M Cache,Turbo, HT, 1066MHz Max Mem
* 8GB Memory (4x2GB), 1333MHz Dual Ranked RDIMMs for 1Processor, Optimized
* 2TB 7.2K RPM SATA
* OS: Windows Server 64bit

Machine B (linux server)
* System Type: Dell PowerEdge R410
* Ship Date: 7/7/2010
* Processor: Intel® Xeon® E5630 2.53Ghz, 12M Cache,Turbo, HT, 1066MHz Max Mem
* 8GB Memory (4x2GB), 1333MHz Dual Ranked RDIMMs for 1Processor, Optimized
* 2TB 7.2K RPM SATA
* OS: Centos 5.5 x86-64

Machine C (DB Host)
* System Type: Gateway E6610D
* Processor, Intel Core2 Duo - E6750 2.66 Ghz
* 4Gb Ram
* 250Gb Hard Drive 7200 rpm
* OS: Centos 5.5 x86-64

Machine D (testing server (JMeter))
* System Type: Dell Precision Workstation 390
* Ship Date: 9/7/2006
* Processor, 6300, 1.86, 2M, Core Duo-conroe, Burn 2
* 2Gb RAM
* 160 Gb Hard drive 7200 rpm
* Service Tag: 5Q5LQB1
* OS: Centos 5.5 i386

== Communication ==

Coordination/communication is primarily via the Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking

Weekly meetings will occur through IRC chat in the #foss4g channel on irc.freenode.net

=== Next IRC meeting ===

* Wed August 18th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=08&day=18&hour=15&min=0&sec=0 15:00:00 UTC]
** Provisional Agenda:
*** baseline test styling equivalence: are we there yet?
*** starting and stopping all servers
*** slotting times for each team to run benchmarks
*** cleaning up the rendering benchmark blank tiles
*** graphic access to the JMeter machine
*** last testing date

=== Previous IRC meeting ===

* Wed August 11th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=08&day=11&hour=15&min=0&sec=0 15:00:00 UTC]
** Provisional Agenda:
*** Jmeter testing
**** what output formats will be requested?
**** what projections will be requested? (Platte-carré is not conformal)
**** what layer combinations will be requested? (All vector layers or various mixed subsets?)
**** what envelopes will be issued? (Frank's 2009 query generation script forces a hard cutoff at the largest scale)
**** what output metrics will be created?
**** how will the jmx file be designed? (need one .csv file per thread so all requests are different)
*** Status report from each team
*** Testing deadline: shall we extend it?
*** Can all server provide links to the sample requests and allow to build more? (style comparison, are we all doing the same work?)
*** Vector data
**** Changes in rendering (missing values for some attributes)
**** Gaps: what to do with the 9 missing sheets?
** Summary: [http://logs.qgis.org/foss4g/%23foss4g.2010-08-11.log log]

* Wed August 4th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=08&day=4&hour=15&min=0&sec=0 15:00:00 UTC]
** Provisional Agenda:
*** Machine status / issues
*** Data status / issues
**** Characters mis-encoded in shapefiles (e.g. catalan in labels): fixable? impact on PostGIS database?
**** Raster conversion to GeoTIFF (torrent of results?)
*** Styles status / issues
*** Jmeter testing
**** what output formats will be requested?
**** what projections will be requested? (Platte-carré is not conformal)
**** what layer combinations will be requested? (All vector layers or various mixed subset?)
**** what envelopes will be issued? (Frank's 2009 query generation script forces a hard cutoff at the largest scale)
**** what output metrics will be created?
**** how will the jmx file be designed? (need one .csv file per thread so all requests are different)
** Summary: [[Benchmarking 2010 IRC meeting logs Aug 4th]]

* Wed July 28th
(no info)

* Wed July 21st @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=07&day=21&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on data processing for all of Spain
*** update on server setup
*** update on data styling
*** checkin on server installs / library dependencies

* Wed July 14th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=07&day=14&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on data processing for all of Spain
*** update on server setup
*** data styling discussions
**** scales to use for layers
**** missing full road network
**** layer to use for labels along lines
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-07-14.log log])
*** Attendance:
**** Jeff McKenna (jmckenna)
**** Andrea Aime (aaime)
**** Iván Sánchez (IvanSanchez)
**** Michael Smith (msmith_)
**** Dimitri Monie (dmonie)
**** Dane Springmeyer (springmeyer)
*** Discussion:
**** data:
***** full dataset of Spain, vector and raster, has been processed by IvanSanchez, and he is checking it and will create a torrent for transfer, by the end of the week
***** because of shapefile size limit some layers may have to be split (contour-1.shp, contour-2.shp)
**** servers:
***** msmith has received both server machines
***** T1 router is being installed, it will be a dedicated line for this project
***** he is adding 2 TB drives to each
**** layer styling:
***** we notice that the Spain dataset we are using (BCN25) does not include street files (zoomed in road network)
***** we agreed to use contours for curved label tests instead
***** jmckenna will test the layer scales provided by Marco (QGIS mapserver)
***** aaime pointed out problem with the posted SLDs, in terms of the label property (jmckenna to look into it)
***** aaime pointed out that we should make sure that all servers compute the scales the same way
****** OGC recommends 90 DPI (GeoServer, Mapnik use the recommended 90...MapServer defaults to 72, but should be configurable)
****** all teams should look into how their engine handles this and report to group
**** other:
***** IvanSanchez reported that a Spanish custom-made WMS server may join the exercise (they access raster files only)
***** dmonie pointed out that Erdas has been quiet lately because one of the team leaders is on holiday

* Wed June 30th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=30&hour=15&min=0&sec=0 15:00:00 UTC]
** Summary: [http://logs.qgis.org/foss4g/%23foss4g.2010-06-30.log log]

* Wed June 30th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=30&hour=15&min=0&sec=0 15:00:00 UTC]
** Summary: [http://logs.qgis.org/foss4g/%23foss4g.2010-06-30.log log]

* Wed June 23rd @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=23&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on proposed layer styling
*** update on available server specs
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-06-23.log log])

* Wed June 16th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=16&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on server infrastructure
*** update on OSGeo-es dataset ([[Benchmarking 2010/How to get some sample data|Sample data download instructions]])
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-06-16.log log])
*** Attendance:
**** Jeff McKenna (jmckenna)
**** Andrea Aime (aaime)
**** Iván Sánchez (IvanSanchez)
**** Michael Smith (msmith_)
**** Cédric Briançon (cedricbr)
*** Discussion:
**** "base" run will use only shapefiles, "best" run will be any format (raster/vector)
**** teams can optionally test Oracle, PostGIS, Ingres connections during the "best" run
**** layers tested will be: labelled roads (with labels that follow the lines), thematic polygon map, and a test with labelled roads + thematic polygon + point layer in a single request (3 separate runs)
***** msmith and jmckenna to examine styling for one sheet (Barcelona, sheet# 420)
**** each base and best run will test data in raw projection (ETRS89 + UTM28/29/30/31) and then reprojected on-the-fly to the Google Mercator projection
**** threads (number of concurrent requests) will be: 1-2-4-8-16-32-64
**** US Army Corps is proposing to provide 4 desktop machines. 2 identical for testing ( win/lin ) and 1 for jmeter and 1 for databases
***** msmith will find out availability of 2-cpu machines this week

* Wed June 2nd @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=02&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** team representative introductions
*** Establish this year's 'Rules of Engagement' ([http://wiki.osgeo.org/wiki/Benchmarking_2009#Rules_of_Engagement previous RoE])
*** Discuss dataset to be used (including how to share the large dataset, for running local tests)
*** Discuss server infrastructure
*** Discuss possible changes to the test script parameters
*** Set next meeting date
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-06-02.log log])
*** Attendance:
**** Jeff McKenna, MapServer (jmckenna)
**** Cédric Briançon - Constellation (cedricbr)
**** Michael Smith - Oracle Mapviewer and Oracle connections in MapServer (msmith_)
**** Zac Spitzer - MapGuide (zacspitzer)
**** Martin Daly - Cadcorp (mpdaly)
**** Anne-Sophie Collignon - ERDAS (ascollignon)
**** Marco Hugentobler - QGIS mapserver (marco___)
**** Andrea Aime - GeoServer (aaime)
**** Dimitri Monie - ERDAS (dmonie)
**** Frank Warmerdam - MapServer, possibly Ingres as well (FrankW)
**** Iván Sánchez - provide geodata from Spanish sources and help out Mapnik (IvanSanchez)
**** Dane Springmeyer - Mapnik (springmeyer)
**** Johann Sorel - GeotoolKit / Constellation (Eclesia)
**** Adrian Custer - Constellation (acuster)
**** Pirmin Kalberer - QGIS mapserver (pirmin_k)
**** Trevor Wekel - MapGuide (trevorw)
**** Daniel Morissette - MapServer (danmo)
*** Discussion:
**** All agreed to this year's "Rules of Engagement"
***** All parties must contribute any changes that they make to their software for this exercise, back to their community. Note that the changes don't have to be contributed before the conference, just in a reasonable period of time.
***** Comparisons will be made of the best available version of the software, be it a formal release or a development version.
***** Two tests will be run: one 'baseline' test with the data in its raw format (with spatial indexes), and another 'best effort' test where 'the sky is the limit' for what changes you want to make to the data (change format, generalize, etc)
***** Teams must document all steps they did to manipulate the data/server for both the 'baseline' and 'best effort' tests. If a team does not document the steps on this wiki then that team's test results will not be used.
***** Data formats to be used will be shapefiles for vectors, and uncompressed geotiffs for rasters.
***** WMS output formats to be used will be png8 and png24 where possible
**** Dataset
***** Iván Sánchez Ortega from OSGeo-es is willing to provide local Spain datasets
***** data is for non-commercial use only
***** 10 GB of shapefiles
***** 100 GB of uncompressed rasters
**** Server infrastructure
***** possible hosts are msmith (US Army Corps) or Skygone (http://www.skygoneinc.com/)
***** need two identical servers, one running Unix and another Windows, plus a client machine
***** specs could be: quad core, 8GB of memory, 1TB hard drive
***** need fast pipe to server, as large datasets will be transferred
**** Next meeting date
***** all agreed to use same day and time next week, but limit the meeting length to one hour

== Sample Dataset Styling ==

* download [http://download.osgeo.org/benchmarking/barcelona-sample-dataset.zip sample dataset]
* layer [http://wiki.osgeo.org/wiki/Benchmarking_2010/How_to_get_some_sample_data#Vector_data_issues styling rules]
{| border="1"
|-
| '''Mapping Server'''
| '''GetMap'''
| '''SLD'''
|-
| MapServer
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 all layers]
| --
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=settlement&SRS=EPSG:25831&BBOX=426633,4579926,429740,4582528&FORMAT=image/png&WIDTH=950&HEIGHT=600 settlement]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/settlement-sld.xml settlement-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=building&SRS=EPSG:25831&BBOX=426633,4579926,429740,4582528&FORMAT=image/png&WIDTH=950&HEIGHT=600 building]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/building-sld.xml building-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=industry&SRS=EPSG:25831&BBOX=424932,4575961,427328,4577812&FORMAT=image/png&WIDTH=950&HEIGHT=600 industry]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/industry-sld.xml industry-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=contour_lines&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 contour lines]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/contour-sld.xml contour-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=motorway&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 motorway]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/motorway-sld.xml motorway-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=road&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 road]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/road-sld.xml road-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=ramp&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 ramp]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/ramp-sld.xml ramp-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=track&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 track]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/track-sld.xml track-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=point_geometry&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 point labels with geometry]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/point-geom-sld.xml point-geom-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=point_nogeom&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 point labels without geometry]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/point-nogeom-sld.xml point-nogeom-sld.xml]
|}

=== Scale dependent rules checking requests ===

These requests show the behavior of scale dependent styling around the critical scales.
The geographic area requested is always the same:

<tt>BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102</tt>

The image size changes instead to alter the scale:

* roughly 1:29k: <tt>WIDTH=824&HEIGHT=472</tt>
* roughly 1:31k: <tt>WIDTH=771&HEIGHT=442</tt>
* roughly 1:49k: <tt>WIDTH=488&HEIGHT=280</tt>
* roughly 1.51k: <tt>WIDTH=469&HEIGHT=269</tt>

The table allows to compare the various servers output. The GeoServer links use a vendor option to get the actual scale, as computed by the server, displayed at the bottom/right of the map.

{| border="1"
|-
| '''Server'''
| '''1:29k'''
| '''1:31k'''
| '''1:49k'''
| '''1:51k'''
|-
| Constellation-SDI
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
| GeoServer
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472&format_options=layout:scale 1:29k]
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442&format_options=layout:scale 1:31k]
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280&format_options=layout:scale 1:49k]
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269&format_options=layout:scale 1:51k]
|-
| Mapnik
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
| MapServer
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
| MapViewer
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
|-
| GeognoSIS
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
|-
| QGIS mapserver
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
|-
| ERDAS APOLLO
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472&srs=EPSG:4326&FORMAT=image/png 1:29k]
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442&srs=EPSG:4326&FORMAT=image/png 1:31k]
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280&height=600&srs=EPSG:4326&FORMAT=image/png 1:49k]
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269&srs=EPSG:4326&FORMAT=image/png 1:51k]
|}

== Benchmarking setup ==

Tests are going to be run using JMeter, using a progression of 1, 2, 4, 8, 16, 32, and 64 threads, each thread group doing 100, 200, 200, 400, 400, 800 requests respectively, for a total of 2200 requests.
The requests bounds and sizes are going to be picked up from a csv file generated by a random generator script.

The scripts to be run for the baseline tests, and repeated for the best effort one, are:
* raster data, JPEG output, EPSG:25831, scales between 1:1M and 1:10k
* raster data, JPEG output, EPSG:3857, scales between 1:1M and 1:10k
* vector data, PNG24 output, EPSG:4326, scales between 1:300k and 1:1k
* vector data, PNG24 output, EPSG:3875, scales between 1:300k and 1:1k
* raster + vector, JPEG output, EPSG:25831, scales between 1:300k and 1:1k
* raster + vector, JPEG output, EPSG:3785, scales between 1:300k and 1:1k

The areas to be tested are:
* 6 E, 38 N, -1 E, 43 N for vector tests
* 372360,4557880,484490,4631460 for raster tests

The raster layer in the baseline test is going to be the mosaic of GeoTIFF files Frank prepared out of the ECW files on the Linux server.

The vector data set is composed of a stack of individual layers to be stacked in the following order (bottom to top):
* settlements
* contour-0
* contour-1
* contour-2
* contour-3
* contour-4
* contour-5
* contour-6
* contour-7
* building
* industry
* track
* ramp
* road
* motorway
* point-labels-for-geometry
* point-labels-no-geometry

The best effort will use its own custom set of layers (e.g., it's likely contours will be merged into one, and so on)

== Live Benchmark WMS GetMap Requests ==

Note: sample bbox's fixed on Aug 12 to fit the proper aspect ratio of an 800/600 image request (springmeyer). Previous bbox for barcelona request was '1.8,41,2.4,42', new one used below is: '1.43333333333,41.0,2.76666666667,42.0'. Previous request for focused view was '2.1076723642349,41.407828508849,2.1178733654021,41.414271246429', new one is '2.10767236423,41.4072245022,2.1178733654,41.4148752531'. This was done to sidestep a deficiency in mapnik for affine transformation of geometries according to the WMS spec when the image w/h aspect does not match the bbox aspect, but was agreed upon as useful for these demo links by jeff and andrea in IRC.

=== MapServer ===

* Windows 64bit Server
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (zoomed-in)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (mid-zoom)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=316295,4519985.6640625,540555,4669784.3359375&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (whole-area)

* Linux
** [http://12.189.158.78:8081/cgi-bin/mapserv?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.78:8081/cgi-bin/mapserv?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev-label-fix] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev-label-fix] (labelled contours, larger view)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (zoomed-in)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (mid-zoom)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=316295,4519985.6640625,540555,4669784.3359375&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (whole-area)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle-raster-only.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg Oracle GeoRaster, MapServer 5.7-dev] (zoomed-in)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle-raster-only.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg Oracle GeoRaster, MapServer 5.7-dev] (mid-zoom)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle-raster-only.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=316295,4519985.6640625,540555,4669784.3359375&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg Oracle GeoRaster, MapServer 5.7-dev] (whole-area)

=== GeoServer ===

* [http://12.189.158.78:8080/geoserver/wms?SRS=EPSG%3A4326&LAYERS=shp_alllayers_label&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles GeoServer 2.1.x] (Barcelona extents)
*[http://12.189.158.78:8080/geoserver/wms?SRS=EPSG%3A4326&LAYERS=shp_alllayers_label&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Shapefiles GeoServer 2.1.x] (with labelled contours)
* [http://12.189.158.78:8080/geoserver/wms?EXCEPTIONS=application%2Fvnd.ogc.se_inimage&LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=2.10767236423,41.4072245022,2.1178733654,41.41487525316&WIDTH=800&HEIGHT=600 Shapefiles GeoServer 2.1.x] (labelled contours, larger view)
* [http://12.189.158.78:8080/geoserver/wms?LAYERS=ras%3Amosaic_tif&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 TIFF mosaic, zoomed in, GeoServer 2.1.x]
* [http://12.189.158.78:8080/geoserver/wms?LAYERS=ras%3Amosaic_tif&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 TIFF mosaic, mid zoom, GeoServer 2.1.x]
* [http://12.189.158.78:8080/geoserver/wms?LAYERS=ras%3Amosaic_tif&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 TIFF mosaic, whole area, GeoServer 2.1.x]

=== Mapnik ===
* [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles Mapnik2 / paleoserver] (Barcelona extents)
* [http://12.189.158.78:8091/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles Mapnik0.7.2 / mod_mapnik_wms] (Barcelona extents)
* [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Shapefiles Mapnik2 / paleoserver] (larger view - labeled contours)

=== Constellation-SDI ===
* [http://12.189.158.78:8280/constellation/WS/wms?SRS=EPSG%3A4326&LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles Constellation] (Barcelona extents)
* [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=2.10767236423,41.4072245022,2.1178733654,41.41487525316&WIDTH=800&HEIGHT=600 Shapefiles Constellation] (labelled contours, larger view)

=== Oracle MapViewer ===
*[http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600&srs=EPSG:4326&FORMAT=image/png&aa=TRUE Oracle DB / MapViewer] (Barcelona Extents)

*[http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png&aa=TRUE Oracle DB / MapViewer] (with labeled contours)

=== Cadcorp GeognoSIS ===
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600 Shapefiles, GeognoSIS 7.0] (Barcelona Extents)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Shapefiles, GeognoSIS 7.0] (with labelled contours)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-postgis&BGCOLOR=0xFFFFFF&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600 PostGIS, GeognoSIS 7.0] (Barcelona Extents)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-postgis&BGCOLOR=0xFFFFFF&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 PostGIS, GeognoSIS 7.0] (with labelled contours)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-oracle&BGCOLOR=0xFFFFFF&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600 Oracle, GeognoSIS 7.0] (Barcelona Extents)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-oracle&BGCOLOR=0xFFFFFF&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Oracle, GeognoSIS 7.0] ('''no contours''' - under investigation)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-tif&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 TIFF mosaic, zoomed in, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-tif&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 TIFF mosaic, mid zoom, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-tif&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 TIFF mosaic, whole area, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-ecw&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 ECW, zoomed in, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-ecw&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 ECW, mid zoom, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-ecw&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 ECW, whole area, GeognoSIS 7.0]

=== QGIS mapserver ===
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SRS=EPSG:4326&LAYERS=shp_vector&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles, QGIS mapserver] (Barcelona Extents)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, QGIS mapserver] (with labeled contours)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/pg.qgs&SRS=EPSG:4326&LAYERS=shp_vector&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 PostGIS, QGIS mapserver] (Barcelona Extents)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/pg.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png PostGIS, QGIS mapserver] (with labeled contours)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector_raster.qgs&LAYERS=vrt&STYLES=&FORMAT=image/jpg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3043&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 TIFF mosaic, zoomed in, QGIS mapserver]
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector_raster.qgs&LAYERS=vrt&STYLES=&FORMAT=image/jpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3043&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 TIFF mosaic, mid zoom, QGIS mapserver]
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector_raster.qgs&LAYERS=vrt&STYLES=&FORMAT=image/jpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3043&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 TIFF mosaic, whole area, QGIS mapserver]

=== ERDAS APOLLO ===

Windows 64bit Server
* [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, APOLLO 10.1.2] (Barcelona extents)
* [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, APOLLO 10.1.2] (with labelled contours)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=TIFF&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 Geotiff, APOLLO 10.1.2] (Zoomed-in)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=TIFF&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 Geotiff, APOLLO 10.1.2](mid-zoom)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=TIFF&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 Geotiff, APOLLO 10.1.2] (Whole Area)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=BENCHMARK&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 ECW, APOLLO 10.1.2] (Zoomed-in)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=BENCHMARK&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 ECW, APOLLO 10.1.2] (mid-zoom)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=BENCHMARK&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 ECW, APOLLO 10.1.2] (Whole Area)

== Starting and stopping the various servers ==

* GeoServer on the Linux box:
** <tt>cd /benchmarking/geoserver</tt>
** starting: <tt>/benchmarking/geoserver/start_geoserver.sh</tt>
** stopping gracefully: <tt>/benchmarking/geoserver/stop_geoserver.sh</tt>
** check it's really dead: <tt>ps aux | grep GEOSERVER</tt> . If it hasn't stopped gracefully use <tt>kill -9 <pid></tt> to terminate the process

* MapServer and QGIS Mapserver on the Linux box (both are running in the same Apache instance):
** starting: /opt/mapserver/bin/apachectl start
** stopping: /opt/mapserver/bin/apachectl stop

* MapServer on Windows box:
** Command prompt (right-click "Run as Administrator")
** cd \ms4w
** apache-stop.bat
** apache-start.bat

* Mapnik:
** paleoserver (standalone daemon):
*** stopping: /opt/mapnik/paleoserver_stop.sh
*** starting: /opt/mapnik/paleoserver_start.sh
** mod_mapnik_wms (runs within apache):
** starting: /opt/mapnik/mod_mapnik_start.sh
** stopping: /opt/mapnik/mod_mapnik_stop.sh

* Mapviewer:
** <tt>cd /benchmarking/mapviewer</tt>
** starting: <tt> /opt/mapviewer/start_mv.sh </tt>
** stopping: <tt> /opt/mapviewer/stop_mv.sh </tt>
** check it's really dead: <tt>ps aux | grep java | grep mapviewer</tt> . If it hasn't stopped gracefully use <tt>kill -9 <pid></tt> to terminate the process

* Cadcorp GeognoSIS
** 7.0
*** Command line (with "Run as Administrator")
**** starting: net start geognosis-70
**** stopping: net stop geognosis-70
*** Or use services.msc, service name geognosis-70
** 7.1
*** Same as 7.0, but with "-71" suffix

* Erdas Windows Server
** APOLLO SDI (vector services)
*** Command Line (with "Run as Administrator")
**** starting : cd C:\opt\ERDAS\APOLLOSDI\tomcat\bin\startup.bat
**** stopping : cd C:\opt\ERDAS\APOLLOSDI\tomcat\bin\shutdown.bat
*** check the java process for erdas user has been killed
** APOLLO IWS (raster services)
*** Open IIS (Control Panel\Administrative Tools --> Internet Information Services (IIS) Manager)
**** starting : select under "Site" "Default Web Site" --> Manage Web Site : Start
**** Stopping : select under "Site" "Default Web Site" --> Manage Web Site : Stop

* Constellation on the Linux box (you must be logged in as constellation user, do not launch it as root !):
** starting: <tt>/opt/constellation/apache-tomcat-6.0.28/bin/startup.sh</tt>
** stopping: <tt>/opt/constellation/apache-tomcat-6.0.28/bin/shutdown.sh</tt>
** check it's really dead: <tt>ps aux | grep constellation</tt> . If it hasn't stopped gracefully use <tt>kill -9 <pid></tt> to terminate the process

* Oracle Database
** Login to database machine and sudo to user oracle
***Starting: lsntctl start; dbstart
***Stopping: lsntcrl stop: dbshut

== List of Software Enhancements ==

=== GeoServer ===

* support DBF larger than 2GB
* avoid reading the Z dimension when flat geometries are all that's needed
* generally improve shapefile reading: don't open the shp and dbf file if the spatial index says there is no data to read, bbox based feature skipping during filtering, limit number of disk reads to the bare minimum
* perform fast rectangular clipping of geometries before sending them to the renderer
* direct rendering path for raster to avoid the java2d bottlenecks
* improved .qix index file generation based on node load factor instead of tree depth, node collapsing for degenerate linear chains
* light parallelization of the renderer so that feature reading and rendering can be performed in parallel
* better metadata caching (feature type structure and the like)

=== MapServer ===
* large DBF support ([https://trac.osgeo.org/mapserver/ticket/3514 ticket])
* improving labels on curved lines ([http://trac.osgeo.org/mapserver/ticket/3523 ticket])
* handling label precision ([http://trac.osgeo.org/mapserver/ticket/2950 ticket])
* improving Oracle GeoRaster GDAL driver ([http://trac.osgeo.org/gdal/ticket/3746 ticket])

=== Constellation-SDI ===

Detailed in the [[Benchmarking_2010/Constellation-SDI|Constellation-SDI]] page.

=== Cadcorp GeognoSIS ===

Detailed in the [[Benchmarking_2010/Cadcorp_notes|Cadcorp notes]] page.

=== ERDAS APOLLO ===

For APOLLO IWS :
* Upgrade to GDAL 1.7.2
* Direct access to GDAL for Geotiff reader
* Improved the mosaicing of TIF datasets to remove seam lines
For APOLLO SDI :
* Tomcat 6 and Java 6: improved performances over Tomcat 5.5 and Java 5 Rendering :
* Clash management for contour labels
* Multipath rendering optimisation
* Low level code optimisation
* New option for shape index in memory

=== Mapnik ===
Detailed at: [[Benchmarking_2010/Mapnik_notes#Improvements]]

== List of Performance Results ==

=== Constellation-SDI ===

Detailed in the [[Benchmarking_2010/Constellation-SDI|Constellation-SDI]] page.

== External Related Links ==

*[[FOSS4G Benchmark|FOSS4G WMS Benchmark]]
*[http://sourceforge.net/projects/wmstester/ WMSTester - tool for testing not from OSGeo]

----

[[Category:FOSS4G2010]] [[Category:FOSS4G]]

Benchmarking 2010

2010-09-23T09:58:25Z

Wiki-Cedricbr: /* Starting and stopping the various servers */

== Basic Premise ==

Following up on [[Benchmarking 2009|last year's exercise]], the performance shoot-out presentation at [[FOSS4G2010]] will test how long each Web mapping server takes to generate a map image, from a common set of spatial data, on a common platform. The data will be served by each Web mapping server through the WMS standard, which will serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to measure various aspects of those layers.

== Participants ==

{| border="1"
|-
| '''Mapping Server'''
| '''Development Team Leader'''
| '''IRC nick'''
| '''Server status'''
|-
| Cadcorp GeognoSIS
| [mailto:martin.daly--@--cadcorp.com Martin Daly]
| mpdaly
| Production release of server installed with preliminary configuration for SHP, PostGIS, TIF and ECW
|-
| Constellation-SDI
| [[User:Acuster|Adrian Custer]] [[User:Cedricbr|Cédric Briançon]]
| acuster cedricbr
| Running on port 8280. Installation done for Shapefiles and Rasters.
|-
| Erdas Apollo
| [mailto:dimitri.monie@erdas.com Dimitri Monie begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting begin_of_the_skype_highlighting end_of_the_skype_highlighting]
| dmonie
| Install and configuration on-going on windows server, ecw WMS configuration done, shape and DB WMS configuration on going. Port 80, port 8080
|-
| GeoServer
| Andrea Aime
| aaime
| Normally running on port 8080 on the Linux server. Shapefiles and TIFF mosaic have been configured.
|-
| <s>MapGuide Open Source</s>
| <s>Zac Spitzer</s>
| <s>zacspitzer</s>
| ''inactive''
|-
| Mapnik
| [[User:Springmeyer|Dane Springmeyer]]
| springmeyer
| Paleoserver is running on port 8090(shape) / 8091(postgis) / 8092(raster - no reprojection). (mod_mapnik_wms is an alternate being tested, will run on same ports if active).
|-
| MapServer
| [[User:Jmckenna|Jeff McKenna]]
| jmckenna
| linux_wms_bm serving WMS requests for shps, on port 8081. windows wms_bm serving requests for shps, on port 8081.
|-
| Oracle MapViewer
| [[User:msmitherdc|Michael Smith]]
| msmith_
| Set up on port 8088
|-
| QGIS mapserver
| Marco Hugentobler
| marco___
| Set up on port 8081 (Linux server). Shapefiles and TIF (without reprojection) configured
|}

== Timeline ==

{| border="1"
|-
| January 1st, 2010
| begin contacting all mapping servers
|-
| June 1st, 2010
| commitment due by all interested mapping servers
|-
| June 2nd, 2010
| exercise begins (and weekly meetings)
|-
| August 1st, 2010
| final testing begins
|-
| August 23rd, 2010
| no further testing
|-
| September 6-9, 2010
| present results at FOSS4G2010
|}

== Rules of Engagement ==

# All parties must contribute any changes that they make to their software for this exercise, back to their community. Note that the changes don't have to be contributed before the conference, just in a reasonable period of time.
# Comparisons will be made of the best available version of the software, be it a formal release or a development version.
# Two tests will be run: one 'baseline' test with the data in its raw format (with spatial indexes), and another 'best effort' test where 'the sky is the limit' for what changes you want to make to the data (change format, generalize, etc)
# Teams must document all steps they did to manipulate the data/server for both the 'baseline' and 'best effort' tests. If a team does not document the steps on this wiki then that team's test results will not be used.
# Data formats to be used will be shapefiles for vectors, and uncompressed geotiffs for rasters.
# WMS output formats to be used will be png8 and png24 where possible

== Documenting Server Details and Differences ==

It is the responsibility of each team to document their setup with regard to data.

Please keep your notes in your servers directory in svn: http://svn.osgeo.org/osgeo/foss4g/benchmarking/

One particular thing that will differ is each team's use of spatial indexes on shapefiles.

Teams are '''not''' allowed to modify the .shp, .shx, .dbf or .prj files of the vector data (for the baseline benchmark). They can, however, create auxiliary files for spatial indexes and the like.

{| border="1"
| '''Mapping Server''' || '''ext''' || '''type''' || '''command to create'''
|-
| MapServer || .qix || quadtree || shptree <shpfile> ([[shptree notes]])
|-
| GeoServer || .qix || quadtree || Using the ones generated for MapServer atm
|-
| Mapnik || .index || quadtree || shapeindex <shpfile> ([[Benchmarking_2010/Mapnik_notes#shpindex_depth]])
|-
|Erdas || .rtr || RTree || RTreeBuilder basedir shapename max min ([[RTreeBuilder notes]])
|-
|Cadcorp || *.shp.idx || RTree || GUI command in desktop SIS ([[Benchmarking_2010/Cadcorp_notes]])
|-
| Constellation-SDI || .qix || quadtree || Using the ones generated by MapServer
|-
| MapViewer || .oix || RTree || GUI option in MapBuilder.jar, java map/theme/style builder
|}

* Note: in your server setup documentation be sure to record the EXACT command used to create the indexes, with appropriate options passed (e.g depth)

== Datasets ==

A set of data published by the spanish National Mapping Agency will be used. This data is free for non-commercial use, so it's perfectly OK to use it in the benchmark.

Vector data will be a topographic map, composed of several shapefiles (one shapefile per theme). Raster data will be 50cm/px aerial photography.

We still do not have the definitive datasets, due to their size and the processing time needed to put them together. You can get some sample data by reading [[Benchmarking 2010/How to get some sample data]].

== SVN ==

The project files (minus data) are stored in Subversion (http://svn.osgeo.org/osgeo/foss4g/benchmarking/).

== Hardware ==

Machine A (windows server)
* System Type: Dell PowerEdge R410
* Ship Date: 7/7/2010
* Processor: Intel® Xeon® E5630 2.53Ghz, 12M Cache,Turbo, HT, 1066MHz Max Mem
* 8GB Memory (4x2GB), 1333MHz Dual Ranked RDIMMs for 1Processor, Optimized
* 2TB 7.2K RPM SATA
* OS: Windows Server 64bit

Machine B (linux server)
* System Type: Dell PowerEdge R410
* Ship Date: 7/7/2010
* Processor: Intel® Xeon® E5630 2.53Ghz, 12M Cache,Turbo, HT, 1066MHz Max Mem
* 8GB Memory (4x2GB), 1333MHz Dual Ranked RDIMMs for 1Processor, Optimized
* 2TB 7.2K RPM SATA
* OS: Centos 5.5 x86-64

Machine C (DB Host)
* System Type: Gateway E6610D
* Processor, Intel Core2 Duo - E6750 2.66 Ghz
* 4Gb Ram
* 250Gb Hard Drive 7200 rpm
* OS: Centos 5.5 x86-64

Machine D (testing server (JMeter))
* System Type: Dell Precision Workstation 390
* Ship Date: 9/7/2006
* Processor, 6300, 1.86, 2M, Core Duo-conroe, Burn 2
* 2Gb RAM
* 160 Gb Hard drive 7200 rpm
* Service Tag: 5Q5LQB1
* OS: Centos 5.5 i386

== Communication ==

Coordination/communication is primarily via the Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking

Weekly meetings will occur through IRC chat in the #foss4g channel on irc.freenode.net

=== Next IRC meeting ===

* Wed August 18th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=08&day=18&hour=15&min=0&sec=0 15:00:00 UTC]
** Provisional Agenda:
*** baseline test styling equivalence: are we there yet?
*** starting and stopping all servers
*** slotting times for each team to run benchmarks
*** cleaning up the rendering benchmark blank tiles
*** graphic access to the JMeter machine
*** last testing date

=== Previous IRC meeting ===

* Wed August 11th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=08&day=11&hour=15&min=0&sec=0 15:00:00 UTC]
** Provisional Agenda:
*** Jmeter testing
**** what output formats will be requested?
**** what projections will be requested? (Platte-carré is not conformal)
**** what layer combinations will be requested? (All vector layers or various mixed subsets?)
**** what envelopes will be issued? (Frank's 2009 query generation script forces a hard cutoff at the largest scale)
**** what output metrics will be created?
**** how will the jmx file be designed? (need one .csv file per thread so all requests are different)
*** Status report from each team
*** Testing deadline: shall we extend it?
*** Can all server provide links to the sample requests and allow to build more? (style comparison, are we all doing the same work?)
*** Vector data
**** Changes in rendering (missing values for some attributes)
**** Gaps: what to do with the 9 missing sheets?
** Summary: [http://logs.qgis.org/foss4g/%23foss4g.2010-08-11.log log]

* Wed August 4th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=08&day=4&hour=15&min=0&sec=0 15:00:00 UTC]
** Provisional Agenda:
*** Machine status / issues
*** Data status / issues
**** Characters mis-encoded in shapefiles (e.g. catalan in labels): fixable? impact on PostGIS database?
**** Raster conversion to GeoTIFF (torrent of results?)
*** Styles status / issues
*** Jmeter testing
**** what output formats will be requested?
**** what projections will be requested? (Platte-carré is not conformal)
**** what layer combinations will be requested? (All vector layers or various mixed subset?)
**** what envelopes will be issued? (Frank's 2009 query generation script forces a hard cutoff at the largest scale)
**** what output metrics will be created?
**** how will the jmx file be designed? (need one .csv file per thread so all requests are different)
** Summary: [[Benchmarking 2010 IRC meeting logs Aug 4th]]

* Wed July 28th
(no info)

* Wed July 21st @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=07&day=21&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on data processing for all of Spain
*** update on server setup
*** update on data styling
*** checkin on server installs / library dependencies

* Wed July 14th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=07&day=14&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on data processing for all of Spain
*** update on server setup
*** data styling discussions
**** scales to use for layers
**** missing full road network
**** layer to use for labels along lines
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-07-14.log log])
*** Attendance:
**** Jeff McKenna (jmckenna)
**** Andrea Aime (aaime)
**** Iván Sánchez (IvanSanchez)
**** Michael Smith (msmith_)
**** Dimitri Monie (dmonie)
**** Dane Springmeyer (springmeyer)
*** Discussion:
**** data:
***** full dataset of Spain, vector and raster, has been processed by IvanSanchez, and he is checking it and will create a torrent for transfer, by the end of the week
***** because of shapefile size limit some layers may have to be split (contour-1.shp, contour-2.shp)
**** servers:
***** msmith has received both server machines
***** T1 router is being installed, it will be a dedicated line for this project
***** he is adding 2 TB drives to each
**** layer styling:
***** we notice that the Spain dataset we are using (BCN25) does not include street files (zoomed in road network)
***** we agreed to use contours for curved label tests instead
***** jmckenna will test the layer scales provided by Marco (QGIS mapserver)
***** aaime pointed out problem with the posted SLDs, in terms of the label property (jmckenna to look into it)
***** aaime pointed out that we should make sure that all servers compute the scales the same way
****** OGC recommends 90 DPI (GeoServer, Mapnik use the recommended 90...MapServer defaults to 72, but should be configurable)
****** all teams should look into how their engine handles this and report to group
**** other:
***** IvanSanchez reported that a Spanish custom-made WMS server may join the exercise (they access raster files only)
***** dmonie pointed out that Erdas has been quiet lately because one of the team leaders is on holiday

* Wed June 30th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=30&hour=15&min=0&sec=0 15:00:00 UTC]
** Summary: [http://logs.qgis.org/foss4g/%23foss4g.2010-06-30.log log]

* Wed June 30th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=30&hour=15&min=0&sec=0 15:00:00 UTC]
** Summary: [http://logs.qgis.org/foss4g/%23foss4g.2010-06-30.log log]

* Wed June 23rd @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=23&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on proposed layer styling
*** update on available server specs
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-06-23.log log])

* Wed June 16th @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=16&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** update on server infrastructure
*** update on OSGeo-es dataset ([[Benchmarking 2010/How to get some sample data|Sample data download instructions]])
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-06-16.log log])
*** Attendance:
**** Jeff McKenna (jmckenna)
**** Andrea Aime (aaime)
**** Iván Sánchez (IvanSanchez)
**** Michael Smith (msmith_)
**** Cédric Briançon (cedricbr)
*** Discussion:
**** "base" run will use only shapefiles, "best" run will be any format (raster/vector)
**** teams can optionally test Oracle, PostGIS, Ingres connections during the "best" run
**** layers tested will be: labelled roads (with labels that follow the lines), thematic polygon map, and a test with labelled roads + thematic polygon + point layer in a single request (3 separate runs)
***** msmith and jmckenna to examine styling for one sheet (Barcelona, sheet# 420)
**** each base and best run will test data in raw projection (ETRS89 + UTM28/29/30/31) and then reprojected on-the-fly to the Google Mercator projection
**** threads (number of concurrent requests) will be: 1-2-4-8-16-32-64
**** US Army Corps is proposing to provide 4 desktop machines. 2 identical for testing ( win/lin ) and 1 for jmeter and 1 for databases
***** msmith will find out availability of 2-cpu machines this week

* Wed June 2nd @ [http://timeanddate.com/worldclock/fixedtime.html?year=2010&month=06&day=02&hour=15&min=0&sec=0 15:00:00 UTC]
** Agenda:
*** team representative introductions
*** Establish this year's 'Rules of Engagement' ([http://wiki.osgeo.org/wiki/Benchmarking_2009#Rules_of_Engagement previous RoE])
*** Discuss dataset to be used (including how to share the large dataset, for running local tests)
*** Discuss server infrastructure
*** Discuss possible changes to the test script parameters
*** Set next meeting date
** Summary: ([http://logs.qgis.org/foss4g/%23foss4g.2010-06-02.log log])
*** Attendance:
**** Jeff McKenna, MapServer (jmckenna)
**** Cédric Briançon - Constellation (cedricbr)
**** Michael Smith - Oracle Mapviewer and Oracle connections in MapServer (msmith_)
**** Zac Spitzer - MapGuide (zacspitzer)
**** Martin Daly - Cadcorp (mpdaly)
**** Anne-Sophie Collignon - ERDAS (ascollignon)
**** Marco Hugentobler - QGIS mapserver (marco___)
**** Andrea Aime - GeoServer (aaime)
**** Dimitri Monie - ERDAS (dmonie)
**** Frank Warmerdam - MapServer, possibly Ingres as well (FrankW)
**** Iván Sánchez - provide geodata from Spanish sources and help out Mapnik (IvanSanchez)
**** Dane Springmeyer - Mapnik (springmeyer)
**** Johann Sorel - GeotoolKit / Constellation (Eclesia)
**** Adrian Custer - Constellation (acuster)
**** Pirmin Kalberer - QGIS mapserver (pirmin_k)
**** Trevor Wekel - MapGuide (trevorw)
**** Daniel Morissette - MapServer (danmo)
*** Discussion:
**** All agreed to this year's "Rules of Engagement"
***** All parties must contribute any changes that they make to their software for this exercise, back to their community. Note that the changes don't have to be contributed before the conference, just in a reasonable period of time.
***** Comparisons will be made of the best available version of the software, be it a formal release or a development version.
***** Two tests will be run: one 'baseline' test with the data in its raw format (with spatial indexes), and another 'best effort' test where 'the sky is the limit' for what changes you want to make to the data (change format, generalize, etc)
***** Teams must document all steps they did to manipulate the data/server for both the 'baseline' and 'best effort' tests. If a team does not document the steps on this wiki then that team's test results will not be used.
***** Data formats to be used will be shapefiles for vectors, and uncompressed geotiffs for rasters.
***** WMS output formats to be used will be png8 and png24 where possible
**** Dataset
***** Iván Sánchez Ortega from OSGeo-es is willing to provide local Spain datasets
***** data is for non-commercial use only
***** 10 GB of shapefiles
***** 100 GB of uncompressed rasters
**** Server infrastructure
***** possible hosts are msmith (US Army Corps) or Skygone (http://www.skygoneinc.com/)
***** need two identical servers, one running Unix and another Windows, plus a client machine
***** specs could be: quad core, 8GB of memory, 1TB hard drive
***** need fast pipe to server, as large datasets will be transferred
**** Next meeting date
***** all agreed to use same day and time next week, but limit the meeting length to one hour

== Sample Dataset Styling ==

* download [http://download.osgeo.org/benchmarking/barcelona-sample-dataset.zip sample dataset]
* layer [http://wiki.osgeo.org/wiki/Benchmarking_2010/How_to_get_some_sample_data#Vector_data_issues styling rules]
{| border="1"
|-
| '''Mapping Server'''
| '''GetMap'''
| '''SLD'''
|-
| MapServer
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 all layers]
| --
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=settlement&SRS=EPSG:25831&BBOX=426633,4579926,429740,4582528&FORMAT=image/png&WIDTH=950&HEIGHT=600 settlement]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/settlement-sld.xml settlement-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=building&SRS=EPSG:25831&BBOX=426633,4579926,429740,4582528&FORMAT=image/png&WIDTH=950&HEIGHT=600 building]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/building-sld.xml building-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=industry&SRS=EPSG:25831&BBOX=424932,4575961,427328,4577812&FORMAT=image/png&WIDTH=950&HEIGHT=600 industry]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/industry-sld.xml industry-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=contour_lines&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 contour lines]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/contour-sld.xml contour-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=motorway&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 motorway]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/motorway-sld.xml motorway-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=road&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 road]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/road-sld.xml road-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=ramp&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 ramp]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/ramp-sld.xml ramp-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=track&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 track]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/track-sld.xml track-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=point_geometry&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 point labels with geometry]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/point-geom-sld.xml point-geom-sld.xml]
|-
|
| [http://labs.gatewaygeomatics.com/cgi-bin/benchmark2010?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=point_nogeom&SRS=EPSG:25831&BBOX=400637.456357,4576080.998882,456535.105266,4594947.942275&FORMAT=image/png&WIDTH=950&HEIGHT=600 point labels without geometry]
| [http://labs.gatewaygeomatics.com/benchmarking/sld/point-nogeom-sld.xml point-nogeom-sld.xml]
|}

=== Scale dependent rules checking requests ===

These requests show the behavior of scale dependent styling around the critical scales.
The geographic area requested is always the same:

<tt>BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102</tt>

The image size changes instead to alter the scale:

* roughly 1:29k: <tt>WIDTH=824&HEIGHT=472</tt>
* roughly 1:31k: <tt>WIDTH=771&HEIGHT=442</tt>
* roughly 1:49k: <tt>WIDTH=488&HEIGHT=280</tt>
* roughly 1.51k: <tt>WIDTH=469&HEIGHT=269</tt>

The table allows to compare the various servers output. The GeoServer links use a vendor option to get the actual scale, as computed by the server, displayed at the bottom/right of the map.

{| border="1"
|-
| '''Server'''
| '''1:29k'''
| '''1:31k'''
| '''1:49k'''
| '''1:51k'''
|-
| Constellation-SDI
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
| GeoServer
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472&format_options=layout:scale 1:29k]
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442&format_options=layout:scale 1:31k]
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280&format_options=layout:scale 1:49k]
| [http://12.189.158.78:8080/geoserver/wms?LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269&format_options=layout:scale 1:51k]
|-
| Mapnik
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
| MapServer
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
| MapViewer
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&srs=EPSG:4326&FORMAT=image/png&aa=TRUE&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
|-
| GeognoSIS
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
|-
| QGIS mapserver
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472 1:29k]
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442 1:31k]
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280 1:49k]
| [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&srs=EPSG:4326&FORMAT=image/png&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269 1:51k]
|-
|-
| ERDAS APOLLO
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=824&HEIGHT=472&srs=EPSG:4326&FORMAT=image/png 1:29k]
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=771&HEIGHT=442&srs=EPSG:4326&FORMAT=image/png 1:31k]
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=488&HEIGHT=280&height=600&srs=EPSG:4326&FORMAT=image/png 1:49k]
| [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=0.58671954345704,40.73922127533,0.64684991455078,40.773670967102&WIDTH=469&HEIGHT=269&srs=EPSG:4326&FORMAT=image/png 1:51k]
|}

== Benchmarking setup ==

Tests are going to be run using JMeter, using a progression of 1, 2, 4, 8, 16, 32, and 64 threads, each thread group doing 100, 200, 200, 400, 400, 800 requests respectively, for a total of 2200 requests.
The requests bounds and sizes are going to be picked up from a csv file generated by a random generator script.

The scripts to be run for the baseline tests, and repeated for the best effort one, are:
* raster data, JPEG output, EPSG:25831, scales between 1:1M and 1:10k
* raster data, JPEG output, EPSG:3857, scales between 1:1M and 1:10k
* vector data, PNG24 output, EPSG:4326, scales between 1:300k and 1:1k
* vector data, PNG24 output, EPSG:3875, scales between 1:300k and 1:1k
* raster + vector, JPEG output, EPSG:25831, scales between 1:300k and 1:1k
* raster + vector, JPEG output, EPSG:3785, scales between 1:300k and 1:1k

The areas to be tested are:
* 6 E, 38 N, -1 E, 43 N for vector tests
* 372360,4557880,484490,4631460 for raster tests

The raster layer in the baseline test is going to be the mosaic of GeoTIFF files Frank prepared out of the ECW files on the Linux server.

The vector data set is composed of a stack of individual layers to be stacked in the following order (bottom to top):
* settlements
* contour-0
* contour-1
* contour-2
* contour-3
* contour-4
* contour-5
* contour-6
* contour-7
* building
* industry
* track
* ramp
* road
* motorway
* point-labels-for-geometry
* point-labels-no-geometry

The best effort will use its own custom set of layers (e.g., it's likely contours will be merged into one, and so on)

== Live Benchmark WMS GetMap Requests ==

Note: sample bbox's fixed on Aug 12 to fit the proper aspect ratio of an 800/600 image request (springmeyer). Previous bbox for barcelona request was '1.8,41,2.4,42', new one used below is: '1.43333333333,41.0,2.76666666667,42.0'. Previous request for focused view was '2.1076723642349,41.407828508849,2.1178733654021,41.414271246429', new one is '2.10767236423,41.4072245022,2.1178733654,41.4148752531'. This was done to sidestep a deficiency in mapnik for affine transformation of geometries according to the WMS spec when the image w/h aspect does not match the bbox aspect, but was agreed upon as useful for these demo links by jeff and andrea in IRC.

=== MapServer ===

* Windows 64bit Server
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (zoomed-in)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (mid-zoom)
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=316295,4519985.6640625,540555,4669784.3359375&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (whole-area)

* Linux
** [http://12.189.158.78:8081/cgi-bin/mapserv?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.78:8081/cgi-bin/mapserv?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev-label-fix] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, MapServer 5.7-dev-label-fix] (labelled contours, larger view)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=postgis_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png PostGIS, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (Barcelona extents)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Oracle, MapServer 5.7-dev] (with labelled contours)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (zoomed-in)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (mid-zoom)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-raster.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=raster_benchmarking_2010&STYLES=&BBOX=316295,4519985.6640625,540555,4669784.3359375&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg GeoTIFF, MapServer 5.7-dev] (whole-area)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle-raster-only.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg Oracle GeoRaster, MapServer 5.7-dev] (zoomed-in)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle-raster-only.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg Oracle GeoRaster, MapServer 5.7-dev] (mid-zoom)
** [http://12.189.158.78:8081/cgi-bin/mapservfull?MAP=/benchmarking/mapserver/spain-oracle-raster-only.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=oracle_benchmarking_2010&STYLES=&BBOX=316295,4519985.6640625,540555,4669784.3359375&width=800&height=600&srs=EPSG:25831&FORMAT=image/jpeg Oracle GeoRaster, MapServer 5.7-dev] (whole-area)

=== GeoServer ===

* [http://12.189.158.78:8080/geoserver/wms?SRS=EPSG%3A4326&LAYERS=shp_alllayers_label&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles GeoServer 2.1.x] (Barcelona extents)
*[http://12.189.158.78:8080/geoserver/wms?SRS=EPSG%3A4326&LAYERS=shp_alllayers_label&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Shapefiles GeoServer 2.1.x] (with labelled contours)
* [http://12.189.158.78:8080/geoserver/wms?EXCEPTIONS=application%2Fvnd.ogc.se_inimage&LAYERS=shp_alllayers_label&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=2.10767236423,41.4072245022,2.1178733654,41.41487525316&WIDTH=800&HEIGHT=600 Shapefiles GeoServer 2.1.x] (labelled contours, larger view)
* [http://12.189.158.78:8080/geoserver/wms?LAYERS=ras%3Amosaic_tif&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 TIFF mosaic, zoomed in, GeoServer 2.1.x]
* [http://12.189.158.78:8080/geoserver/wms?LAYERS=ras%3Amosaic_tif&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 TIFF mosaic, mid zoom, GeoServer 2.1.x]
* [http://12.189.158.78:8080/geoserver/wms?LAYERS=ras%3Amosaic_tif&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 TIFF mosaic, whole area, GeoServer 2.1.x]

=== Mapnik ===
* [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles Mapnik2 / paleoserver] (Barcelona extents)
* [http://12.189.158.78:8091/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles Mapnik0.7.2 / mod_mapnik_wms] (Barcelona extents)
* [http://12.189.158.78:8090/?SRS=EPSG%3A4326&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Shapefiles Mapnik2 / paleoserver] (larger view - labeled contours)

=== Constellation-SDI ===
* [http://12.189.158.78:8280/constellation/WS/wms?SRS=EPSG%3A4326&LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles Constellation] (Barcelona extents)
* [http://12.189.158.78:8280/constellation/WS/wms?LAYERS=shp:settlement,shp:contour-0,shp:contour-1,shp:contour-2,shp:contour-3,shp:contour-4,shp:contour-5,shp:contour-6,shp:contour-7,shp:building,shp:industry,shp:track,shp:ramp,shp:road,shp:motorway,shp:point-labels-for-geometry,shp:point-labels-no-geometry&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=2.10767236423,41.4072245022,2.1178733654,41.41487525316&WIDTH=800&HEIGHT=600 Shapefiles Constellation] (labelled contours, larger view)

=== Oracle MapViewer ===
*[http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600&srs=EPSG:4326&FORMAT=image/png&aa=TRUE Oracle DB / MapViewer] (Barcelona Extents)

*[http://12.189.158.78:8088/mapviewer/wms?REQUEST=GetMap&BASEMAP=spain_oracle&LAYERS=&SERVICE=WMS&VERSION=1.1.1&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png&aa=TRUE Oracle DB / MapViewer] (with labeled contours)

=== Cadcorp GeognoSIS ===
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600 Shapefiles, GeognoSIS 7.0] (Barcelona Extents)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-shp&BGCOLOR=0xFFFFFF&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Shapefiles, GeognoSIS 7.0] (with labelled contours)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-postgis&BGCOLOR=0xFFFFFF&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600 PostGIS, GeognoSIS 7.0] (Barcelona Extents)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-postgis&BGCOLOR=0xFFFFFF&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 PostGIS, GeognoSIS 7.0] (with labelled contours)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-oracle&BGCOLOR=0xFFFFFF&BBOX=1.43333333333,41.0,2.76666666667,42.0&width=800&height=600 Oracle, GeognoSIS 7.0] (Barcelona Extents)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=spain-oracle&BGCOLOR=0xFFFFFF&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600 Oracle, GeognoSIS 7.0] ('''no contours''' - under investigation)
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-tif&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 TIFF mosaic, zoomed in, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-tif&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 TIFF mosaic, mid zoom, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-tif&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 TIFF mosaic, whole area, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-ecw&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 ECW, zoomed in, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-ecw&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 ECW, mid zoom, GeognoSIS 7.0]
* [http://12.189.158.75/cadcorp/geognosis-70.exe?VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&STYLES=&FORMAT=image%2fpng%3bPhotometricInterpretation=RGB&SRS=EPSG:4326&LAYERS=bcn-ecw&BGCOLOR=0xFFFFFF&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 ECW, whole area, GeognoSIS 7.0]

=== QGIS mapserver ===
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SRS=EPSG:4326&LAYERS=shp_vector&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 Shapefiles, QGIS mapserver] (Barcelona Extents)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, QGIS mapserver] (with labeled contours)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/pg.qgs&SRS=EPSG:4326&LAYERS=shp_vector&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600 PostGIS, QGIS mapserver] (Barcelona Extents)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/pg.qgs&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shp_vector&STYLES=&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png PostGIS, QGIS mapserver] (with labeled contours)
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector_raster.qgs&LAYERS=vrt&STYLES=&FORMAT=image/jpg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3043&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 TIFF mosaic, zoomed in, QGIS mapserver]
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector_raster.qgs&LAYERS=vrt&STYLES=&FORMAT=image/jpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3043&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 TIFF mosaic, mid zoom, QGIS mapserver]
* [http://12.189.158.78:8081/fcgi-bin/qgis_mapserv.fcgi?MAP=/opt/qgisms/fcgi-bin/vector_raster.qgs&LAYERS=vrt&STYLES=&FORMAT=image/jpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3043&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 TIFF mosaic, whole area, QGIS mapserver]

=== ERDAS APOLLO ===

Windows 64bit Server
* [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=1.43333333333,41.0,2.76666666667,42.0&WIDTH=800&HEIGHT=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, APOLLO 10.1.2] (Barcelona extents)
* [http://12.189.158.75:8080/erdas-apollo/vector/FOSS4GSHAPES?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=shape_vector&STYLES=prop&BBOX=2.10767236423,41.4072245022,2.1178733654,41.4148752531&width=800&height=600&srs=EPSG:4326&FORMAT=image/png Shapefiles, APOLLO 10.1.2] (with labelled contours)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=TIFF&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 Geotiff, APOLLO 10.1.2] (Zoomed-in)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=TIFF&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 Geotiff, APOLLO 10.1.2](mid-zoom)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=TIFF&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 Geotiff, APOLLO 10.1.2] (Whole Area)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=BENCHMARK&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=449647.84729003,4594968.4096903,450085.85510253,4595260.9852213&WIDTH=512&HEIGHT=342 ECW, APOLLO 10.1.2] (Zoomed-in)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=BENCHMARK&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=441127.2265625,4590970.3051752,455143.4765625,4600332.7221674&WIDTH=512&HEIGHT=342 ECW, APOLLO 10.1.2] (mid-zoom)
* [http://12.189.158.75/ImageX/ecw_wms.dll?LAYERS=BENCHMARK&STYLES=&FORMAT=image%2Fjpeg&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A25831&BBOX=316295,4519985.6640625,540555,4669784.3359375&WIDTH=512&HEIGHT=342 ECW, APOLLO 10.1.2] (Whole Area)

== Starting and stopping the various servers ==

* GeoServer on the Linux box:
** <tt>cd /benchmarking/geoserver</tt>
** starting: <tt>/benchmarking/geoserver/start_geoserver.sh</tt>
** stopping gracefully: <tt>/benchmarking/geoserver/stop_geoserver.sh</tt>
** check it's really dead: <tt>ps aux | grep GEOSERVER</tt> . If it hasn't stopped gracefully use <tt>kill -9 <pid></tt> to terminate the process

* MapServer and QGIS Mapserver on the Linux box (both are running in the same Apache instance):
** starting: /opt/mapserver/bin/apachectl start
** stopping: /opt/mapserver/bin/apachectl stop

* MapServer on Windows box:
** Command prompt (right-click "Run as Administrator")
** cd \ms4w
** apache-stop.bat
** apache-start.bat

* Mapnik:
** paleoserver (standalone daemon):
*** stopping: /opt/mapnik/paleoserver_stop.sh
*** starting: /opt/mapnik/paleoserver_start.sh
** mod_mapnik_wms (runs within apache):
** starting: /opt/mapnik/mod_mapnik_start.sh
** stopping: /opt/mapnik/mod_mapnik_stop.sh

* Mapviewer:
** <tt>cd /benchmarking/mapviewer</tt>
** starting: <tt> /opt/mapviewer/start_mv.sh </tt>
** stopping: <tt> /opt/mapviewer/stop_mv.sh </tt>
** check it's really dead: <tt>ps aux | grep java | grep mapviewer</tt> . If it hasn't stopped gracefully use <tt>kill -9 <pid></tt> to terminate the process

* Cadcorp GeognoSIS
** 7.0
*** Command line (with "Run as Administrator")
**** starting: net start geognosis-70
**** stopping: net stop geognosis-70
*** Or use services.msc, service name geognosis-70
** 7.1
*** Same as 7.0, but with "-71" suffix

* Erdas Windows Server
** APOLLO SDI (vector services)
*** Command Line (with "Run as Administrator")
**** starting : cd C:\opt\ERDAS\APOLLOSDI\tomcat\bin\startup.bat
**** stopping : cd C:\opt\ERDAS\APOLLOSDI\tomcat\bin\shutdown.bat
*** check the java process for erdas user has been killed
** APOLLO IWS (raster services)
*** Open IIS (Control Panel\Administrative Tools --> Internet Information Services (IIS) Manager)
**** starting : select under "Site" "Default Web Site" --> Manage Web Site : Start
**** Stopping : select under "Site" "Default Web Site" --> Manage Web Site : Stop

* Constellation on the Linux box (you must be logged in as constellation user):
** starting: <tt>/opt/constellation/apache-tomcat-6.0.28/bin/startup.sh</tt>
** stopping: <tt>/opt/constellation/apache-tomcat-6.0.28/bin/shutdown.sh</tt>
** check it's really dead: <tt>ps aux | grep constellation</tt> . If it hasn't stopped gracefully use <tt>kill -9 <pid></tt> to terminate the process

* Oracle Database
** Login to database machine and sudo to user oracle
***Starting: lsntctl start; dbstart
***Stopping: lsntcrl stop: dbshut

== List of Software Enhancements ==

=== GeoServer ===

* support DBF larger than 2GB
* avoid reading the Z dimension when flat geometries are all that's needed
* generally improve shapefile reading: don't open the shp and dbf file if the spatial index says there is no data to read, bbox based feature skipping during filtering, limit number of disk reads to the bare minimum
* perform fast rectangular clipping of geometries before sending them to the renderer
* direct rendering path for raster to avoid the java2d bottlenecks
* improved .qix index file generation based on node load factor instead of tree depth, node collapsing for degenerate linear chains
* light parallelization of the renderer so that feature reading and rendering can be performed in parallel
* better metadata caching (feature type structure and the like)

=== MapServer ===
* large DBF support ([https://trac.osgeo.org/mapserver/ticket/3514 ticket])
* improving labels on curved lines ([http://trac.osgeo.org/mapserver/ticket/3523 ticket])
* handling label precision ([http://trac.osgeo.org/mapserver/ticket/2950 ticket])
* improving Oracle GeoRaster GDAL driver ([http://trac.osgeo.org/gdal/ticket/3746 ticket])

=== Constellation-SDI ===

Detailed in the [[Benchmarking_2010/Constellation-SDI|Constellation-SDI]] page.

=== Cadcorp GeognoSIS ===

Detailed in the [[Benchmarking_2010/Cadcorp_notes|Cadcorp notes]] page.

=== ERDAS APOLLO ===

For APOLLO IWS :
* Upgrade to GDAL 1.7.2
* Direct access to GDAL for Geotiff reader
* Improved the mosaicing of TIF datasets to remove seam lines
For APOLLO SDI :
* Tomcat 6 and Java 6: improved performances over Tomcat 5.5 and Java 5 Rendering :
* Clash management for contour labels
* Multipath rendering optimisation
* Low level code optimisation
* New option for shape index in memory

=== Mapnik ===
Detailed at: [[Benchmarking_2010/Mapnik_notes#Improvements]]

== List of Performance Results ==

=== Constellation-SDI ===

Detailed in the [[Benchmarking_2010/Constellation-SDI|Constellation-SDI]] page.

== External Related Links ==

*[[FOSS4G Benchmark|FOSS4G WMS Benchmark]]
*[http://sourceforge.net/projects/wmstester/ WMSTester - tool for testing not from OSGeo]

----

[[Category:FOSS4G2010]] [[Category:FOSS4G]]

Benchmarking 2010/Constellation-SDI

2010-09-03T14:31:55Z

Wiki-Cedricbr: /* Geotoolkit */

This page describes the experience of the Constellation-SDI team during the FOSS4G 2010 Benchmarking effort.

== Benchmark design ==

We expended a great deal of effort attempting to understand how a benchmarking effort could be designed properly.

The 2009 and 2010 efforts were undertaken in the naive belief that one simply sets up different servers on the same data and makes requests of the servers to compare response times. Belatedly, the 2010 effort is demonstrating that a proper benchmark is more complex, since the data might be in a format useful for a certain class of use cases but meaningless for another scale of usage, the testing can easily be hardware bound limiting useful comparison between servers, servers can be set up to be doing very different work especially in 'best effort'/anything goes configurations, and results can be compared only in superficial ways or in very narrow types of requests. In order to tackle these issues rather than merely pretend they were not serious, we examined what it would take to develop a useful benchmarking protocol, either to stress all the functionality of one particular server or to compare the performance and abilities of various, arbitrary WMS servers.

Developing a WMS benchmarking design which provides useful, comparative metrics of server performance is exceedingly hard. In a recent presentation at the Java Language Summit, Joshua Bloch presented a talk entitled [http://wiki.jvmlangsummit.com/images/1/1d/PerformanceAnxiety2010.pdf ''Performance Anxiety''] which describes the impossibility of developing performant software from first principles in any language due to the enhancements of compilation and machine instruction re-ordering, the necessity of testing to obtain concrete results, and the difficulty of developing proper, statistically rigourous testing metrics.

Since these issues were apparent to us even before this effort, we have been developing tools, benchmark designs and analytic methodologies to test the Constellation-SDI server. This work has been greatly extended during the FOSS4G 2010 benchmarking effort and expanded to consider how to test different WMS servers, possibly built for different uses.

Unfortunately, there is still much distance to go before achieving a solid benchmarking suite. This work will undoubtedly be continued in the future, most likely within the framework of Open Geospatial Consortium (OGC) testing.

== Enhancements ==

This section describes enhancements due to the work during the benchmarking 2010 effort, including improved understanding and workflow by the Constellation-SDI team and ameliorations to the code bases of the [http://www.geotoolkit.org/ Geotoolkit.org] library and to the [http://www.constellation-sdi.org Constellation-SDI] server itself.

=== Benchmarking ===

* Investigate numerous issues with jmeter.
* Design simpler scripts.
* Examine different configurations to stress different aspects of the WMS server experience.

=== Geotoolkit ===

* Referencing: fix inverse projection for fake spherical mercator.
* Referencing: accelerate raster reprojection.

* Coverage: Create a reader for GeoTiff images.

* Shapefile: reduce memory usage when leveraging a quad-tree index.
* Shapefile: reduce styling to one single pass when painting by symbol rather than by feature.
* Shapefile: reduce the reading of non-necessary parts of the files.

* DataSource: Enable startup from a coverage mosaic, either a folder or a manager.

* Renderer: bypass rendering engine for single raster requests.
* Renderer: improve decimation algorithm for vector layers.
* Renderer: switch to OGC conformant 96dpi assumption rather than industry standard 72.
* Renderer: fix sld parsing errors to handle <ogc:Literal> or its absence.
* Renderer: optimize the colour model selected for multiple inputs.

=== Constellation-SDI ===

* Configuration: greatly enhance configurability of the server, with hot reload of data, styles and rendering configuration.

* Server: cache ServiceMetadata document for insanely slow data sources.
* Server: fix envelopes for data sources in ServiceMetadata document.

* Backend: fix multi-threading bug to use classes in a thread-safe manner.

* JEE output: enable direct writing of images into output stream.

* GUI: build a prototype interface.

== Performance Results ==

This section details the results of running the jmeter scripts against the Constellation-SDI WMS server.

'''Note:''' All values reported are in units of responses per second taken from the "Throughput" column of the 'summarizer.py' script. The values are those of the third pass, after the jmeter scripts have looped through the two warmup passes for the various thread counts, from one to sixty-four, and then we measure the last pass.

=== Session 2010.09.03 ===

This was the first full run of Constellation-SDI using the jmeter scripts.

The runs were performed with the more recent jmeter design where all three runs use different requests so that the server is always asking for new files from disk.

For lack of time, a second run was only done for the two raster request sets; nonetheless, the numbers give us a ballpark estimate of variability between runs.

{| border="1" style="text-align:center"
! colspan ="9" | Raster Results for 2010.09.03
|-
! rowspan ="2" | Threads !! colspan="4" | 25831 !! colspan="4" | 3857
|-
! Run 1 !! Run 2 !! Run 3 !! Run 4 !! Run 1 !! Run 2 !! Run 3 !! Run 4
|-
| 1 || 4.5 || 5.7 || --- || --- || 5.4 || 5.4 || --- || ---
|-
| 2 || 6.3 || 6.1 || --- || --- || 6.0 || 5.7 || --- || ---
|-
| 4 || 5.1 || 4.9 || --- || --- || 4.7 || 4.8 || --- || ---
|-
| 8 || 4.8 || 4.6 || --- || --- || 4.6 || 4.5 || --- || ---
|-
| 16 || 4.5 || 4.7 || --- || --- || 4.6 || 4.6 || --- || ---
|-
| 32 || 5.2 || 5.0 || --- || --- || 4.8 || 4.8 || --- || ---
|-
| 64 || 4.9 || 4.8 || --- || --- || 4.8 || 4.6 || --- || ---
|}

{| border="1" style="text-align:center"
! colspan ="9" | Vector Results for 2010.09.03
|-
! rowspan ="2" | Threads !! colspan="4" | 4326 !! colspan="4" | 3857
|-
! Run 1 !! Run 2 !! Run 3 !! Run 4 !! Run 1 !! Run 2 !! Run 3 !! Run 4
|-
| 1 || 1.5 || --- || --- || --- || 1.5 || --- || --- || ---
|-
| 2 || 2.1 || --- || --- || --- || 2.1 || --- || --- || ---
|-
| 4 || 2.1 || --- || --- || --- || 2.3 || --- || --- || ---
|-
| 8 || 2.2 || --- || --- || --- || 2.3 || --- || --- || ---
|-
| 16 || 2.1 || --- || --- || --- || 2.2 || --- || --- || ---
|-
| 32 || 2.2 || --- || --- || --- || 2.3 || --- || --- || ---
|-
| 64 || 1.8 || --- || --- || --- || 1.9 || --- || --- || ---
|}

The processing power (8 CPUs) of the machine does not seem to have been stressed at any point in the test runs.

The numbers differ in numbers generated on local servers which had a twice as big separation between vector and raster results. Variability also seems high enough that several runs would be needed to discriminate between the various configurations, enough so that we probably need to tighten up the testing to get anything meaningful from numbers such as these.

=== Session NEXT ===

''This is a placeholder and template for future runs''

{| border="1" style="text-align:center"
! colspan ="9" | Raster Results
|-
! rowspan ="2" | Threads !! colspan="4" | 25831 !! colspan="4" | 3857
|-
! Run 1 !! Run 2 !! Run 3 !! Run 4 !! Run 1 !! Run 2 !! Run 3 !! Run 4
|-
| 1 || --- || --- || --- || --- || --- || --- || --- || ---
|-
| 2 || --- || --- || --- || --- || --- || --- || --- || ---
|-
| 4 || --- || --- || --- || --- || --- || --- || --- || ---
|-
| 8 || --- || --- || --- || --- || --- || --- || --- || ---
|-
| 16 || --- || --- || --- || --- || --- || --- || --- || ---
|-
| 32 || --- || --- || --- || --- || --- || --- || --- || ---
|-
| 64 || --- || --- || --- || --- || --- || --- || --- || ---
|}

{| border="1" style="text-align:center"
! colspan ="9" | Vector Results for 2010.09.03
|-
! rowspan ="2" | Threads !! colspan="4" | 4326 !! colspan="4" | 3857
|-
! Run 1 !! Run 2 !! Run 3 !! Run 4 !! Run 1 !! Run 2 !! Run 3 !! Run 4
|-
| 1 || --- || --- || --- || --- || --- || --- || --- || ---
|-
| 2 || --- || --- || --- || --- || --- || --- || --- || ---
|-
| 4 || --- || --- || --- || --- || --- || --- || --- || ---
|-
| 8 || --- || --- || --- || --- || --- || --- || --- || ---
|-
| 16 || --- || --- || --- || --- || --- || --- || --- || ---
|-
| 32 || --- || --- || --- || --- || --- || --- || --- || ---
|-
| 64 || --- || --- || --- || --- || --- || --- || --- || ---
|}

Benchmarking 2010

2010-08-16T12:34:03Z

Wiki-Cedricbr: /* Constellation */

Benchmarking 2010

2010-08-16T10:22:35Z

Wiki-Cedricbr: /* Scale dependent rules checking requests */

Benchmarking 2010

2010-08-16T10:00:34Z

Wiki-Cedricbr: /* Constellation */

Benchmarking 2010

2010-08-13T14:17:29Z

Wiki-Cedricbr: /* Live Benchmark WMS GetMap Requests */

Benchmarking 2010

2010-08-09T16:23:56Z

Wiki-Cedricbr: /* Documenting Server Details and Differences */

Benchmarking 2010

2010-08-09T16:18:33Z

Wiki-Cedricbr: /* Participants */

User:Cedricbr

2010-06-02T13:04:07Z

Wiki-Cedricbr: /* Cédric Briançon */

== Cédric Briançon ==

Projects active member: Geotoolkit.org, Constellation.

Current role: lead developer on WMS and WCS part of Constellation. Implied in XML (un)marshalling using JAXB for Geotoolkit.org.

Was also collaborating on: GeoTools, Geoserver.

Mail: cedric.briancon@geomatys.fr

User:Cedricbr

2010-06-02T13:03:42Z

Wiki-Cedricbr: /* Cédric Briançon */

== Cédric Briançon ==

Projects active member: Geotoolkit.org, Constellation.

Current role: lead developer on WMS and WCS part of Constellation. Implied in XML (un)marshalling using JAXB for Geotoolkit.org.

Was collaborating on: GeoTools, Geoserver.

Mail: cedric.briancon@geomatys.fr

Benchmarking 2010

2010-06-02T13:01:16Z

Wiki-Cedricbr:

User:Cedricbr

2010-06-02T12:59:29Z

Wiki-Cedricbr:

== Cédric Briançon ==

Projects active member: Geotoolkit.org, Constellation.

Current role: lead developer on WMS and WCS part of Constellation. Implied in XML (un)marshalling using JAXB for Geotoolkit.org.

Also collaborator on: GeoTools, Geoserver.

Mail: cedric.briancon@geomatys.fr

User:Cedricbr

2010-06-02T12:58:10Z

Wiki-Cedricbr: Created page with ' == Cédric Briançon == Projects active member: Geotoolkit.org, Constellation. Current role: lead developer on WMS and WCS part of Constellation. Implied in XML (un)marshalling…'

== Cédric Briançon ==

Projects active member: Geotoolkit.org, Constellation.
Current role: lead developer on WMS and WCS part of Constellation. Implied in XML (un)marshalling using JAXB for Geotoolkit.org.

Also collaborator on: GeoTools, Geoserver.

Mail: cedric.briancon@geomatys.fr