Difference between revisions of "Benchmarking 2011"

From OSGeo
Jump to navigation Jump to search
 
(82 intermediate revisions by 15 users not shown)
Line 1: Line 1:
 
== Basic Premise  ==
 
== Basic Premise  ==
  
−
Following up on [[Benchmarking 2010|last year's exercise]], the performance shoot-out presentation at [[FOSS4G2011]] will test how long each Web mapping server takes to generate a map image, from a common set of spatial data, on a common platform. The data will be served by each Web mapping server through the WMS standard, which will serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to measure various aspects of those layers.  
+
Following up on the [[Benchmarking 2010|2010 exercise]], the performance shoot-out presentation at [[FOSS4G 2011]] tested how long each Web mapping server takes to generate a map image, from a common set of spatial data, on a common platform. The data was served by each Web mapping server through the WMS standard, which served exactly the same set of LAYERS. A JMeter load ran on the testing box to measure various aspects of those layers.
  
−
== Potential Participants  ==
+
== Communication  ==
 +
 
 +
Coordination/communication is primarily via the Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking
 +
 
 +
Weekly meetings will occur through IRC chat in the #foss4g channel on irc.freenode.net (you can use [http://webchat.freenode.net/ webchat] to connect in browser)
 +
 
 +
=== Next IRC Meeting ===
 +
 
 +
 
 +
* Wed August 3rd, 2011 @ [http://timeanddate.com/worldclock/fixedtime.html?year=2011&month=08&day=03&hour=14&min=0&sec=0 14:00:00 UTC]
 +
** Provisional Agenda:
 +
*** Status of OSM vector styling from all teams
 +
*** Discuss rasters to be used
 +
*** Discuss testing methodology
 +
 
 +
=== Previous IRC Meetings ===
 +
 
 +
* [[Benchmarking_2011/MeetingLog | Meeting Log]]
 +
 
 +
== Participants  ==
 +
 
 +
''Server Teams:''
  
 
{| border="1"
 
{| border="1"
Line 9: Line 30:
 
| '''Mapping Server'''  
 
| '''Mapping Server'''  
 
| '''Development Team Leader'''
 
| '''Development Team Leader'''
−
| '''Contacted'''
 
 
| '''Confirmed'''
 
| '''Confirmed'''
 
|-
 
|-
 
| Cadcorp GeognoSIS  
 
| Cadcorp GeognoSIS  
 
| [[User:Mpd|Martin Daly]]
 
| [[User:Mpd|Martin Daly]]
−
| Dec 9th, 2010
 
 
| Yes
 
| Yes
 
|-
 
|-
 
| Constellation-SDI  
 
| Constellation-SDI  
−
| [[User:Acuster|Adrian Custer]], [[User:Cedricbr|Cédric Briançon]]
+
| [[User:Desruisseaux|Martin Desruisseaux]], [[User:Eclesia|Johann Sorel]]
−
|  
+
| Yes
−
|
 
−
|-
 
−
| Erdas Apollo
 
−
| [mailto:dimitri.monie@erdas.com Dimitri Monie]
 
−
|
 
−
|
 
−
|-
 
−
| ESRI ArcServer
 
−
| Satish Sankaran
 
−
|
 
−
|
 
 
|-
 
|-
 
| GeoServer  
 
| GeoServer  
−
| Andrea Aime
+
| Andrea Aime, [[User:groldan|Gabriel Roldan]]
−
|
+
| Yes
−
|
 
−
|-
 
−
| MapGuide Open Source
 
−
|
 
−
|
 
−
|  
 
 
|-
 
|-
 
| Mapnik  
 
| Mapnik  
 
| [[User:Springmeyer|Dane Springmeyer]]
 
| [[User:Springmeyer|Dane Springmeyer]]
−
| Dec 8th, 2010
 
 
| Yes
 
| Yes
 
|-
 
|-
 
| MapServer  
 
| MapServer  
 
| [[User:Jmckenna|Jeff McKenna]],  [[User:msmitherdc|Michael Smith]]
 
| [[User:Jmckenna|Jeff McKenna]],  [[User:msmitherdc|Michael Smith]]
−
| Dec 8th, 2010
 
 
| Yes
 
| Yes
 
|-
 
|-
−
| Oracle MapViewer
+
| QGIS Server
−
| LJ Qian
+
| [[User:Pirmin_Kalberer|Pirmin Kalberer]], Marco Hugentobler
−
|  
+
| Yes
−
|
+
|}
 +
 
 +
''Data Teams:''
 +
 
 +
{| border="1"
 +
|-
 +
| '''Data Package'''
 +
| '''Team Leader'''
 +
| '''Comment'''
 
|-
 
|-
−
| QGIS mapserver
+
| [http://imposm.org/docs/imposm/latest/ Imposm]
−
| Marco Hugentobler
+
| Oliver Tonnhofer
−
|  
+
| tool to be used to import OSM data
−
|  
 
 
|-
 
|-
 
| SPOT imagery
 
| SPOT imagery
 
| [[User:Jeffaudi|Jean-Francois (Jeff) Faudi]]
 
| [[User:Jeffaudi|Jean-Francois (Jeff) Faudi]]
−
| Dec 12th, 2010
+
|  
−
| Yes
+
|}
 +
 
 +
''Not Participating:''
 +
 
 +
{| border="1"
 +
|-
 +
| '''Mapping Server'''
 +
| '''Development Team Leader'''
 +
| '''Comment'''
 +
|-
 +
| Erdas Apollo
 +
| [mailto:dimitri.monie@erdas.com Dimitri Monie]
 +
| Response from Luc Donea: Unable to participate
 +
|-
 +
| ESRI ArcServer
 +
| Satish Sankaran
 +
| discussing internally
 +
|-
 +
| MapGuide Open Source
 +
| TBD (contacted mapguide-internals)
 +
| Response from Jason Birch: Unable to participate
 
|}
 
|}
  
Line 81: Line 108:
 
| exercise begins (and weekly meetings)
 
| exercise begins (and weekly meetings)
 
|-
 
|-
−
| August 1st, 2011  
+
| June 1st, 2011  
−
| final testing begins
+
| final testing begins (no more changes to data/styles/hardware, but changes to software is allowed)
 
|-
 
|-
 
| September 1st, 2011  
 
| September 1st, 2011  
Line 94: Line 121:
 
|}
 
|}
  
−
== Rules of Engagement (unofficial, copied from 2010) ==
+
== Rules of Engagement (DRAFT) ==
  
 
# All parties must contribute any changes that they make to their software for this exercise, back to their community. Note that the changes don't have to be contributed before the conference, just in a reasonable period of time.
 
# All parties must contribute any changes that they make to their software for this exercise, back to their community. Note that the changes don't have to be contributed before the conference, just in a reasonable period of time.
 
# Comparisons will be made of the best available version of the software, be it a formal release or a development version.  
 
# Comparisons will be made of the best available version of the software, be it a formal release or a development version.  
−
# Two tests will be run: one 'baseline' test with the data in its raw format (with spatial indexes), and another 'best effort' test where 'the sky is the limit' for what changes you want to make to the data (change format, generalize, etc)
+
# One test will be run: a 'baseline' test with the data in any format desired, but teams cannot generalize or change the data's resolution from its raw values.
−
# Teams must document all steps they did to manipulate the data/server for both the 'baseline' and 'best effort' tests.  If a team does not document the steps on this wiki then that team's test results will not be used.
+
# Teams must document all steps they did to manipulate the data/server (such as spatial indexes created etc.).  If a team does not document the steps on this wiki then that team's test results will not be used.  
−
# Data formats to be used will be shapefiles for vectors, and uncompressed geotiffs for rasters.
 
 
# WMS output formats to be used will be png8 and png24 where possible
 
# WMS output formats to be used will be png8 and png24 where possible
  
 +
== Documenting Server Setup ==
 +
 +
It is the responsibility of each team to document their setup with regard to configuration details, setup notes, and differences between other servers in how data might be accessed/indexed.
 +
 +
Notes for each server:
 +
 +
* [[Benchmarking_2011/Mapnik_notes | Mapnik]]
 +
* [[Benchmarking_2011/Cadcorp_notes | Cadcorp]]
 +
* [[Benchmarking_2011/MapServer_notes | MapServer]]
 +
* [[Benchmarking_2011/QGIS_notes | QGIS Server]]
 +
 +
Stylesheets and scripts in svn using a directory named wms/{year}/{servername}:
 +
 +
* [http://svn.osgeo.org/osgeo/foss4g/benchmarking/wms/2011/mapnik Mapnik]
 +
* [http://svn.osgeo.org/osgeo/foss4g/benchmarking/wms/2011/cadcorp Cadcorp]
 +
* [http://svn.osgeo.org/osgeo/foss4g/benchmarking/wms/2011/mapserver MapServer]
 +
* [http://svn.osgeo.org/osgeo/foss4g/benchmarking/wms/2011/qgis QGIS Server]
 +
 +
== Testing Tool ==
 +
 +
JMeter is used since it can read a list of bbox's via CSV, and the machine that applies load has been updated with Jmeter 2.5 (the latest stable release at the time of the benchmark): http://www.reverse.net/pub/apache//jakarta/jmeter/binaries/jakarta-jmeter-2.5.zip
 +
 +
Jmeter can be run from the command line like:
 +
 +
    /home/jmeterusr/jakarta-jmeter-2.5/bin/jmeter
 +
 +
== Server Ports ==
  
−
== Documenting Server Details and Differences ==
+
Each server has access to a range of ports for testing:
  
−
It is the responsibility of each team to document their setup with regard to data.
+
Mapserver - 8080 - 8089
  
−
Please keep your notes either on a wiki subpage or in svn (or both).
+
Mapnik - 8090 - 8099
  
−
Generally teams keep notes on setup on the wiki while they put stylesheets and scripts in svn.
+
Cadcorp - 4326 - 4335
  
−
A good organization for wiki pages is to create a subpage like: http://wiki.osgeo.org/wiki/Benchmarking_2011/Mapnik_notes
+
Constellation - 8100 - 8109
  
−
And create a directory in svn named {servername}/{year} like: http://svn.osgeo.org/osgeo/foss4g/benchmarking/mapnik/2011
+
GeoServer - 8110 - 8119
  
 
== Datasets ==
 
== Datasets ==
 +
 +
=== OSM Vectors ===
 +
* [[Benchmarking_2011/DB_Schema]]
 +
* [[Benchmarking_2011/Imposm]]
 +
 +
CSV Files created with
 +
 +
<code> ./run_wms_request.sh -count 2200 -region -109 37 -102 41 -minsize 64 64 -maxsize 1024 768 -minres "2.5e-06" -maxres 0.000755 -srs 4326 -srs2 3857 -filter_within colorado.shp </code>
 +
 +
Sample styling:
 +
 +
* [http://12.189.158.78:8081/cgi-bin/mapserv6?MAP=/benchmarking/wms/2011/mapserver/osm-google.map&layers=default&mode=map&map_imagetype=png&mapext=-11690811.758434+4826833.1534376+-11687378.549609+4830266.3622626&imgext=-11690811.758434+4826833.1534376+-11687378.549609+4830266.3622626&map_size=800+800&imgx=400&imgy=400&imgxy=800+800 mapserv6]
 +
** [http://12.189.158.78:8081/cgi-bin/mapserv6?MAP=/benchmarking/wms/2011/mapserver/osm-google.map&mode=browse&template=openlayers&layers=default MapServer Linux (with OpenLayers)]
 +
** [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/wms/2011/mapserver/osm-google.map&mode=browse&template=openlayers&layers=default MapServer Win64 (with OpenLayers)]
 +
 +
* [http://12.189.158.78:8091/?SRS=EPSG:3857&LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&BBOX=-11690811.758434,4826833.1534376,-11687378.549609,4830266.3622626&WIDTH=800&HEIGHT=800 mapnik]
 +
 +
=== DEM Hillshading ===
 +
 +
Sample styling:
 +
 +
* [http://12.189.158.78:8081/cgi-bin/mapserv6?MAP=/benchmarking/wms/2011/mapserver/hill-google.map&layers=hillshade&mode=map&map_imagetype=png&mapext=-11947611.487019+4524873.854257+-11508160.757419+4964324.583857&imgext=-11947611.487019+4524873.854257+-11508160.757419+4964324.583857&map_size=800+800&imgx=400&imgy=400&imgxy=800+800 MapServer 6 Hillshade]
 +
 +
 +
=== DEM Hillshading + OSM Vectors  ===
 +
 +
* MapServer 6 Hillshade + OSM Vectors - [http://12.189.158.78:8081/cgi-bin/mapserv6?MAP=/benchmarking/wms/2011/mapserver/hill-google.map&layers=hillshade%20default&mode=map&map_imagetype=png&mapext=-11741344.300813+4772064.889657+-11631481.618413+4881927.572057&imgext=-11741344.300813+4772064.889657+-11631481.618413+4881927.572057&map_size=800+800&imgx=400&imgy=400&imgxy=800+800 Linux] [http://12.189.158.75:8081/cgi-bin/mapserv.exe?MAP=C:/benchmarking/wms/2011/mapserver/hill-google.map&layers=hillshade%20default&mode=map&map_imagetype=png&mapext=-11741344.300813+4772064.889657+-11631481.618413+4881927.572057&imgext=-11741344.300813+4772064.889657+-11631481.618413+4881927.572057&map_size=800+800&imgx=400&imgy=400&imgxy=800+800 Win64]
 +
 +
OSM Vector Style changes:
 +
  LandN layers turned off (they provide background color)
 +
  LanduseN layers set to 70% opacity
 +
 +
==Running JMeter==
 +
 +
JMeter is run using VNC (port 5901) on the Jmeter server.
 +
 +
 +
 +
==JMeter Results==
 +
 +
There are 2 scripts for summarizing and ploting the JMeter results from benchmarking
 +
 +
1). opt/scripts/2011/jmeter/charts/plotter.py
 +
Takes 3 arguments
 +
  1). The jmeter output
 +
  2). A String to put on the title of the chart
 +
  3). Normal or seed to determine how many thread loops
 +
 +
Example usage:
 +
 +
<code>/opt/scripts/2011/jmeter/charts/plotter.py jmeter_summary_vector-3857-linux-fcgi.txt "MapServer Linux Fast CGI 3857" normal</code>
 +
 +
<code>/opt/scripts/2011/jmeter/charts/plotter.py jmeter_summary_vector_hill_seed_fcgi_3857.txt "MapServer Linux Seed Hillshade Fast CGI 3857" seed</code>
 +
 +
2). /opt/scripts/2011/jmeter/charts/summarizer.py
 +
 +
This will produced the summary tables from the Jmeter results
 +
 +
Example usage:
 +
 +
<code>/opt/scripts/2011/jmeter/charts/summarizer.py jmeter_summary_vector-3857-linux-cgi.txt > ms_vector-3857-linux-cgi.txt</code>
  
 
== SVN ==
 
== SVN ==
  
−
The project files (minus data) are stored in Subversion (http://svn.osgeo.org/osgeo/foss4g/benchmarking/).
+
* The project files (minus data) are stored in Subversion (http://svn.osgeo.org/osgeo/foss4g/benchmarking/wms/2011/). If you need commit access contact jmckenna on IRC.
 +
 
 +
* Data are only stored on the server at '/benchmarking/wms/2011/data/'
 +
 
 +
* ''existing committers can add new OSGeoID users through this page: https://www.osgeo.org/cgi-bin/auth/ldap_group.py?group=osgeo''
  
 
== Hardware ==
 
== Hardware ==
 +
 +
Contact [User:msmitherdc|Michael Smith] with any questions about this hardware or for login credentials.
  
 
windows_wms_bm  (windows server)
 
windows_wms_bm  (windows server)
Line 140: Line 260:
 
* OS: Centos 5.5 x86-64
 
* OS: Centos 5.5 x86-64
  
−
== Communication  ==
+
== Final Results ==
−
 
 
−
Coordination/communication is primarily via the Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking
 
−
 
 
−
Weekly meetings will occur through IRC chat in the #foss4g channel on irc.freenode.net
 
  
−
=== Next IRC meeting ===
+
* [http://svn.osgeo.org/osgeo/foss4g/benchmarking/wms/2011/presentation/ Final presentation slides] (contains charts and results)
−
 
+
* [http://svn.osgeo.org/osgeo/foss4g/benchmarking/wms/2011/jmeter/ Raw results] are stored in the team's folder
−
* Wed January 12th, 2011 @ [http://timeanddate.com/worldclock/fixedtime.html?year=2011&month=01&day=12&hour=15&min=0&sec=0 15:00:00 UTC]
 
−
** Provisional Agenda:
 
−
***
 
  
 
== External Related Links  ==
 
== External Related Links  ==
Line 161: Line 274:
 
----
 
----
  
−
[[Category:FOSS4G2011]] [[Category:FOSS4G]]
+
[[Category:FOSS4G2011]]
 +
[[Category:Benchmarking]]

Latest revision as of 03:15, 24 February 2015

Basic Premise

Following up on the 2010 exercise, the performance shoot-out presentation at FOSS4G 2011 tested how long each Web mapping server takes to generate a map image, from a common set of spatial data, on a common platform. The data was served by each Web mapping server through the WMS standard, which served exactly the same set of LAYERS. A JMeter load ran on the testing box to measure various aspects of those layers.

Communication

Coordination/communication is primarily via the Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking

Weekly meetings will occur through IRC chat in the #foss4g channel on irc.freenode.net (you can use webchat to connect in browser)

Next IRC Meeting

  • Wed August 3rd, 2011 @ 14:00:00 UTC
    • Provisional Agenda:
      • Status of OSM vector styling from all teams
      • Discuss rasters to be used
      • Discuss testing methodology

Previous IRC Meetings

Participants

Server Teams:

Mapping Server Development Team Leader Confirmed
Cadcorp GeognoSIS Martin Daly Yes
Constellation-SDI Martin Desruisseaux, Johann Sorel Yes
GeoServer Andrea Aime, Gabriel Roldan Yes
Mapnik Dane Springmeyer Yes
MapServer Jeff McKenna, Michael Smith Yes
QGIS Server Pirmin Kalberer, Marco Hugentobler Yes

Data Teams:

Data Package Team Leader Comment
Imposm Oliver Tonnhofer tool to be used to import OSM data
SPOT imagery Jean-Francois (Jeff) Faudi

Not Participating:

Mapping Server Development Team Leader Comment
Erdas Apollo Dimitri Monie Response from Luc Donea: Unable to participate
ESRI ArcServer Satish Sankaran discussing internally
MapGuide Open Source TBD (contacted mapguide-internals) Response from Jason Birch: Unable to participate

Timeline

January 1st, 2011 begin contacting all mapping servers
March 1st, 2011 commitment due by all interested mapping servers
March 2nd, 2011 exercise begins (and weekly meetings)
June 1st, 2011 final testing begins (no more changes to data/styles/hardware, but changes to software is allowed)
September 1st, 2011 no further testing
September 2nd, 2011 final results due from all teams
September 12-16, 2011 present results at FOSS4G2011

Rules of Engagement (DRAFT)

  1. All parties must contribute any changes that they make to their software for this exercise, back to their community. Note that the changes don't have to be contributed before the conference, just in a reasonable period of time.
  2. Comparisons will be made of the best available version of the software, be it a formal release or a development version.
  3. One test will be run: a 'baseline' test with the data in any format desired, but teams cannot generalize or change the data's resolution from its raw values.
  4. Teams must document all steps they did to manipulate the data/server (such as spatial indexes created etc.). If a team does not document the steps on this wiki then that team's test results will not be used.
  5. WMS output formats to be used will be png8 and png24 where possible

Documenting Server Setup

It is the responsibility of each team to document their setup with regard to configuration details, setup notes, and differences between other servers in how data might be accessed/indexed.

Notes for each server:

Stylesheets and scripts in svn using a directory named wms/{year}/{servername}:

Testing Tool

JMeter is used since it can read a list of bbox's via CSV, and the machine that applies load has been updated with Jmeter 2.5 (the latest stable release at the time of the benchmark): http://www.reverse.net/pub/apache//jakarta/jmeter/binaries/jakarta-jmeter-2.5.zip

Jmeter can be run from the command line like: 

    /home/jmeterusr/jakarta-jmeter-2.5/bin/jmeter

Server Ports

Each server has access to a range of ports for testing:

Mapserver - 8080 - 8089

Mapnik - 8090 - 8099

Cadcorp - 4326 - 4335

Constellation - 8100 - 8109

GeoServer - 8110 - 8119

Datasets

OSM Vectors

CSV Files created with

./run_wms_request.sh -count 2200 -region -109 37 -102 41 -minsize 64 64 -maxsize 1024 768 -minres "2.5e-06" -maxres 0.000755 -srs 4326 -srs2 3857 -filter_within colorado.shp

Sample styling:

DEM Hillshading

Sample styling:


DEM Hillshading + OSM Vectors

OSM Vector Style changes: 

 LandN layers turned off (they provide background color)
 LanduseN layers set to 70% opacity

Running JMeter

JMeter is run using VNC (port 5901) on the Jmeter server.


JMeter Results

There are 2 scripts for summarizing and ploting the JMeter results from benchmarking

1). opt/scripts/2011/jmeter/charts/plotter.py Takes 3 arguments 

 1). The jmeter output
 2). A String to put on the title of the chart
 3). Normal or seed to determine how many thread loops

Example usage:

/opt/scripts/2011/jmeter/charts/plotter.py jmeter_summary_vector-3857-linux-fcgi.txt "MapServer Linux Fast CGI 3857" normal

/opt/scripts/2011/jmeter/charts/plotter.py jmeter_summary_vector_hill_seed_fcgi_3857.txt "MapServer Linux Seed Hillshade Fast CGI 3857" seed

2). /opt/scripts/2011/jmeter/charts/summarizer.py

This will produced the summary tables from the Jmeter results

Example usage:

/opt/scripts/2011/jmeter/charts/summarizer.py jmeter_summary_vector-3857-linux-cgi.txt > ms_vector-3857-linux-cgi.txt

SVN

  • Data are only stored on the server at '/benchmarking/wms/2011/data/'

Hardware

Contact [User:msmitherdc|Michael Smith] with any questions about this hardware or for login credentials.

windows_wms_bm (windows server)

  • System Type: Dell PowerEdge R410
  • Ship Date: 7/7/2010
  • Processor: Intel® Xeon® E5630 2.53Ghz, 12M Cache,Turbo, HT, 1066MHz Max Mem
  • 8GB Memory (4x2GB), 1333MHz Dual Ranked RDIMMs for 1Processor, Optimized
  • 2TB 7.2K RPM SATA
  • OS: Windows Server 64bit

linux_wms_bm (linux server)

  • System Type: Dell PowerEdge R410
  • Ship Date: 7/7/2010
  • Processor: Intel® Xeon® E5630 2.53Ghz, 12M Cache,Turbo, HT, 1066MHz Max Mem
  • 8GB Memory (4x2GB), 1333MHz Dual Ranked RDIMMs for 1Processor, Optimized
  • 2TB 7.2K RPM SATA
  • OS: Centos 5.5 x86-64

Final Results

External Related Links

Retrieved from "http://wiki.osgeo.org/w/index.php?title=Benchmarking_2011&oldid=81951"