Benchmarking 2011

Basic Premise

Following up on last year's exercise, the performance shoot-out presentation at FOSS4G2011 will test how long each Web mapping server takes to generate a map image, from a common set of spatial data, on a common platform. The data will be served by each Web mapping server through the WMS standard, which will serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to measure various aspects of those layers.

Communication

Coordination/communication is primarily via the Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking

Weekly meetings will occur through IRC chat in the #foss4g channel on irc.freenode.net (you can use webchat to connect in browser)

Next IRC Meeting

Wed April 20th, 2011 @ 14:00:00 UTC
- Provisional Agenda:
  - Create plan for vector (OSM) and raster (DEM) data processing
  - Discuss testing methodology

Previous IRC Meetings

Meeting Log

Potential Participants

Mapping Server	Development Team Leader	Contacted	Confirmed	Comment
Cadcorp GeognoSIS	Martin Daly	Dec 9th, 2010	Yes
Constellation-SDI	Adrian Custer, Cédric Briançon
Erdas Apollo	Dimitri Monie
ESRI ArcServer	Satish Sankaran			discussing internally
GeoServer	Andrea Aime, Gabriel Roldan		Yes
MapGuide Open Source	TBD (contacted mapguide-internals)	Jan 12th, 2011		Response from Jason Birch: Unable to participate
Mapnik	Dane Springmeyer	Dec 8th, 2010	Yes
MapServer	Jeff McKenna, Michael Smith	Dec 8th, 2010	Yes
Oracle MapViewer	LJ Qian		Yes
QGIS Server	Pirmin Kalberer, Marco Hugentobler		Yes
SPOT imagery	Jean-Francois (Jeff) Faudi	Dec 12th, 2010
XtraServer

Timeline

January 1st, 2011	begin contacting all mapping servers
March 1st, 2011	commitment due by all interested mapping servers
March 2nd, 2011	exercise begins (and weekly meetings)
June 1st, 2011	final testing begins (no more changes to data/styles/hardware, but changes to software is allowed)
September 1st, 2011	no further testing
September 2nd, 2011	final results due from all teams
September 12-16, 2011	present results at FOSS4G2011

Rules of Engagement (DRAFT)

All parties must contribute any changes that they make to their software for this exercise, back to their community. Note that the changes don't have to be contributed before the conference, just in a reasonable period of time.
Comparisons will be made of the best available version of the software, be it a formal release or a development version.
One test will be run: a 'baseline' test with the data in any format desired, but teams cannot generalize or change the data's resolution from its raw values.
Teams must document all steps they did to manipulate the data/server (such as spatial indexes created etc.). If a team does not document the steps on this wiki then that team's test results will not be used.
WMS output formats to be used will be png8 and png24 where possible

Documenting Server Details and Differences

It is the responsibility of each team to document their setup with regard to data.

Please keep your notes either on a wiki subpage or in svn (or both).

Generally teams keep notes on setup on the wiki while they put stylesheets and scripts in svn.

A good organization for wiki pages is to create a subpage like: http://wiki.osgeo.org/wiki/Benchmarking_2011/Mapnik_notes

And create a directory in svn named {servername}/{year} like: http://svn.osgeo.org/osgeo/foss4g/benchmarking/mapnik/2011

Datasets

SVN

The project files (minus data) are stored in Subversion (http://svn.osgeo.org/osgeo/foss4g/benchmarking/).

Hardware

windows_wms_bm (windows server)

System Type: Dell PowerEdge R410
Ship Date: 7/7/2010
Processor: Intel® Xeon® E5630 2.53Ghz, 12M Cache,Turbo, HT, 1066MHz Max Mem
8GB Memory (4x2GB), 1333MHz Dual Ranked RDIMMs for 1Processor, Optimized
2TB 7.2K RPM SATA
OS: Windows Server 64bit

linux_wms_bm (linux server)

System Type: Dell PowerEdge R410
Ship Date: 7/7/2010
Processor: Intel® Xeon® E5630 2.53Ghz, 12M Cache,Turbo, HT, 1066MHz Max Mem
8GB Memory (4x2GB), 1333MHz Dual Ranked RDIMMs for 1Processor, Optimized
2TB 7.2K RPM SATA
OS: Centos 5.5 x86-64