Difference between revisions of "Benchmarking 2010"

From OSGeo
Jump to navigation Jump to search
(→‎Participants: IRC nicks)
Line 91: Line 91:
** Agenda:
** Agenda:
*** update on server infrastructure
*** update on server infrastructure
*** update on OSGeo-es dataset
*** update on OSGeo-es dataset ([[Benchmarking 2010/How to get some sample data|Sample data download instructions]])
=== Previous IRC meeting ===
=== Previous IRC meeting ===

Revision as of 06:42, 9 June 2010

Basic Premise

Following up on last year's exercise, the performance shoot-out presentation at FOSS4G2010 will test how long each Web mapping server takes to generate a map image, from a common set of spatial data, on a common platform. The data will be served by each Web mapping server through the WMS standard, which will serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to measure various aspects of those layers.


Mapping Server Development Team Leader IRC nick
Cadcorp GeognoSIS Martin Daly mpdaly
Constellation-SDI Cédric Briançon cedricbr
Erdas Apollo Dimitri Monie dmonie
GeoServer Andrea Aime aaime
MapGuide Open Source Zac Spitzer zacspitzer
Mapnik Dane Springmeyer springmeyer
MapServer Jeff McKenna jmckenna
Oracle MapViewer Michael Smith msmith_
QGIS mapserver Marco Hugentobler marco___


January 1st, 2010 begin contacting all mapping servers
June 1st, 2010 commitment due by all interested mapping servers
June 2nd, 2010 exercise begins (and weekly meetings)
August 1st, 2010 final testing begins
August 23rd, 2010 no further testing
September 6-9, 2010 present results at FOSS4G2010


A set of data published by the spanish National Mapping Agency will be used. This data is free for non-commercial use, so it's perfectly OK to use it in the benchmark.

Vector data will be a topographic map, composed of several shapefiles (one shapefile per theme). Raster data will be 50cm/px aerial photography.

We still do not have the definitive datasets, due to their size and the processing time needed to put them together. You can get some sample data by reading Benchmarking 2010/How to get some sample data.


Coordination/communication is primarily via the Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking

Weekly meetings will occur through IRC chat in the #foss4g channel on irc.freenode.net

Next IRC meeting

Previous IRC meeting

  • Wed June 2nd @ 15:00:00 UTC
    • Agenda:
      • team representative introductions
      • Establish this year's 'Rules of Engagement' (previous RoE)
      • Discuss dataset to be used (including how to share the large dataset, for running local tests)
      • Discuss server infrastructure
      • Discuss possible changes to the test script parameters
      • Set next meeting date
    • Summary: (log)
      • Attendance:
        • Jeff McKenna, MapServer (jmckenna)
        • Cédric Briançon - Constellation (cedricbr)
        • Michael Smith - Oracle Mapviewer and Oracle connections in MapServer (msmith_)
        • Zac Spitzer - MapGuide (zacspitzer)
        • Martin Daly - Cadcorp (mpdaly)
        • Anne-Sophie Collignon - ERDAS (ascollignon)
        • Marco Hugentobler - QGIS mapserver (marco___)
        • Andrea Aime - GeoServer (aaime)
        • Dimitri Monie - ERDAS (dmonie)
        • Frank Warmerdam - MapServer, possibly Ingres as well (FrankW)
        • Iván Sánchez - provide geodata from Spanish sources and help out Mapnik (IvanSanchez)
        • Dane Springmeyer - Mapnik (springmeyer)
        • Johann Sorel - GeotoolKit / Constellation (Eclesia)
        • Adrian Custer - Constellation (acuster)
        • Pirmin Kalberer - QGIS mapserver (pirmin_k)
        • Trevor Wekel - MapGuide (trevorw)
        • Daniel Morissette - MapServer (danmo)
      • Discussion:
        • All agreed to this year's "Rules of Engagement"
          • All parties must contribute any changes that they make to their software for this exercise, back to their community. Note that the changes don't have to be contributed before the conference, just in a reasonable period of time.
          • Comparisons will be made of the best available version of the software, be it a formal release or a development version.
          • Two tests will be run: one 'baseline' test with the data in its raw format (with spatial indexes), and another 'best effort' test where 'the sky is the limit' for what changes you want to make to the data (change format, generalize, etc)
          • Teams must document all steps they did to manipulate the data/server for both the 'baseline' and 'best effort' tests. If a team does not document the steps on this wiki then that team's test results will not be used.
          • Data formats to be used will be shapefiles for vectors, and uncompressed geotiffs for rasters.
          • WMS output formats to be used will be png8 and png24 where possible
        • Dataset
          • Iván Sánchez Ortega from OSGeo-es is willing to provide local Spain datasets
          • data is for non-commercial use only
          • 10 GB of shapefiles
          • 100 GB of uncompressed rasters
        • Server infrastructure
          • possible hosts are msmith (US Army Corps) or Skygone (http://www.skygoneinc.com/)
          • need two identical servers, one running Unix and another Windows, plus a client machine
          • specs could be: quad core, 8GB of memory, 1TB hard drive
          • need fast pipe to server, as large datasets will be transferred
        • Next meeting date
          • all agreed to use same day and time next week, but limit the meeting length to one hour

External Related Links