Difference between revisions of "Benchmarking 2011"
Sourcepole (talk | contribs) (QGIS mapserver team leader) |
Sourcepole (talk | contribs) |
||
Line 87: | Line 87: | ||
|- | |- | ||
| QGIS mapserver | | QGIS mapserver | ||
− | | [[User:Pirmin_Kalberer]], Marco Hugentobler | + | | [[User:Pirmin_Kalberer|Pirmin Kalberer]], Marco Hugentobler |
| | | | ||
| | | |
Revision as of 23:55, 7 March 2011
Basic Premise
Following up on last year's exercise, the performance shoot-out presentation at FOSS4G2011 will test how long each Web mapping server takes to generate a map image, from a common set of spatial data, on a common platform. The data will be served by each Web mapping server through the WMS standard, which will serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to measure various aspects of those layers.
Communication
Coordination/communication is primarily via the Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking
Weekly meetings will occur through IRC chat in the #foss4g channel on irc.freenode.net (you can use webchat to connect in browser)
Next IRC Meeting
- Wed February 23rd, 2011 @ 15:00:00 UTC
- Provisional Agenda:
- Create plan for vector (OSM) and raster (DEM) data processing
- Discuss testing methodology
- final presentation discussion
- Provisional Agenda:
Previous IRC Meetings
Potential Participants
Mapping Server | Development Team Leader | Contacted | Confirmed | Comment |
Cadcorp GeognoSIS | Martin Daly | Dec 9th, 2010 | Yes | |
Constellation-SDI | Adrian Custer, Cédric Briançon | |||
Erdas Apollo | Dimitri Monie | |||
ESRI ArcServer | Satish Sankaran | discussing internally | ||
GeoServer | Andrea Aime, Gabriel Roldan | Yes | ||
MapGuide Open Source | TBD (contacted mapguide-internals) | Jan 12th, 2011 | Response from Jason Birch: Unable to participate | |
Mapnik | Dane Springmeyer | Dec 8th, 2010 | Yes | |
MapServer | Jeff McKenna, Michael Smith | Dec 8th, 2010 | Yes | |
Oracle MapViewer | LJ Qian | Yes | ||
QGIS mapserver | Pirmin Kalberer, Marco Hugentobler | |||
SPOT imagery | Jean-Francois (Jeff) Faudi | Dec 12th, 2010 | ||
XtraServer |
Timeline
January 1st, 2011 | begin contacting all mapping servers |
March 1st, 2011 | commitment due by all interested mapping servers |
March 2nd, 2011 | exercise begins (and weekly meetings) |
June 1st, 2011 | final testing begins (no more changes to data/styles/hardware, but changes to software is allowed) |
September 1st, 2011 | no further testing |
September 2nd, 2011 | final results due from all teams |
September 12-16, 2011 | present results at FOSS4G2011 |
Rules of Engagement (unofficial, copied from 2010)
- All parties must contribute any changes that they make to their software for this exercise, back to their community. Note that the changes don't have to be contributed before the conference, just in a reasonable period of time.
- Comparisons will be made of the best available version of the software, be it a formal release or a development version.
- Two tests will be run: one 'baseline' test with the data in its raw format (with spatial indexes), and another 'best effort' test where 'the sky is the limit' for what changes you want to make to the data (change format, generalize, etc)
- Teams must document all steps they did to manipulate the data/server for both the 'baseline' and 'best effort' tests. If a team does not document the steps on this wiki then that team's test results will not be used.
- Data formats to be used will be shapefiles for vectors, and uncompressed geotiffs for rasters.
- WMS output formats to be used will be png8 and png24 where possible
Documenting Server Details and Differences
It is the responsibility of each team to document their setup with regard to data.
Please keep your notes either on a wiki subpage or in svn (or both).
Generally teams keep notes on setup on the wiki while they put stylesheets and scripts in svn.
A good organization for wiki pages is to create a subpage like: http://wiki.osgeo.org/wiki/Benchmarking_2011/Mapnik_notes
And create a directory in svn named {servername}/{year} like: http://svn.osgeo.org/osgeo/foss4g/benchmarking/mapnik/2011
Datasets
SVN
The project files (minus data) are stored in Subversion (http://svn.osgeo.org/osgeo/foss4g/benchmarking/).
Hardware
windows_wms_bm (windows server)
- System Type: Dell PowerEdge R410
- Ship Date: 7/7/2010
- Processor: Intel® Xeon® E5630 2.53Ghz, 12M Cache,Turbo, HT, 1066MHz Max Mem
- 8GB Memory (4x2GB), 1333MHz Dual Ranked RDIMMs for 1Processor, Optimized
- 2TB 7.2K RPM SATA
- OS: Windows Server 64bit
linux_wms_bm (linux server)
- System Type: Dell PowerEdge R410
- Ship Date: 7/7/2010
- Processor: Intel® Xeon® E5630 2.53Ghz, 12M Cache,Turbo, HT, 1066MHz Max Mem
- 8GB Memory (4x2GB), 1333MHz Dual Ranked RDIMMs for 1Processor, Optimized
- 2TB 7.2K RPM SATA
- OS: Centos 5.5 x86-64
External Related Links
- FOSS4G2010 Benchmarking Presentation
- FOSS4G2009 Benchmarking Presentation
- FOSS4G WMS Benchmark
- WMSTester - tool for testing not from OSGeo