Benchmarking 2010
Basic Premise
Following up on last year's exercise, the performance shoot-out presentation at FOSS4G2010 will test how long each Web mapping server takes to generate a map image, from a common set of spatial data, on a common platform. The data will be served by each Web mapping server through the WMS standard, which will serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to measure various aspects of those layers.
Timeline
| January 1st, 2010 | begin contacting all mapping servers |
| June 1st, 2010 | commitment due by all interested mapping servers |
| June 2nd, 2010 | exercise begins (and weekly meetings) |
| August 1st, 2010 | final testing begins |
| August 23rd, 2010 | no further testing |
| September 6-9, 2010 | present results at FOSS4G2010 |
Communication
Coordination/communication is primarily via the Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking
Weekly meetings will occur through IRC chat in the #foss4g channel on irc.freenode.net
Next IRC meeting
- Wed June 2nd @ 15:00:00 UTC
- Agenda:
- team representative introductions
- Establish this year's 'Rules of Engagement' (previous RoE)
- Discuss dataset to be used (including how to share the large dataset, for running local tests)
- Discuss server infrastructure
- Discuss possible changes to the test script parameters
- Set next meeting date
- Agenda:
Potential Participants
| Mapping Server | Volunteer to contact Dev Team | Contacted Server's Development Team | Confirmed by Server's Development Team | Development Team Leader |
| Cadcorp GeognoSIS | Martin Daly | yes | yes | Martin Daly |
| Constellation-SDI | Adrian Custer | yes | yes | Cédric Briançon |
| Deegree | Simone Giannecchini | yes | ||
| Erdas Apollo | Anne-Sophie Collignon |
yes |
yes |
Dimitri Monie |
| ESRI ArcGIS Server | Jeff McKenna | yes |
"unable to secure the level of resources required" |
|
| GeoServer | Andrea Aime | yes | yes | Andrea Aime |
| Luciad Web Services Suite | Leif Gruenwoldt | yes | ||
| MapGuide Open Source | Daniel Morissette | yes | yes | Zac Spitzer |
| Mapnik | Dane Springmeyer | yes | yes | Dane Springmeyer |
| MapServer | Jeff McKenna | yes | yes | Jeff McKenna |
| Oracle MapViewer | Michael Smith | yes | yes | Michael Smith |