Difference between revisions of "Benchmarking 2009"

From OSGeo
Jump to navigation Jump to search
 
(145 intermediate revisions by 10 users not shown)
Line 1: Line 1:
 +
== Overview ==
 +
 +
{| border="1"
 +
|Description
 +
|The performance shoot-out will see mapping servers compared in terms of how long they take to generate a map image, from common set of spatial data.
 +
|-
 +
|Participants
 +
|development teams from GeoServer, MapServer
 +
|-
 +
|Results Announced
 +
|[http://2009.foss4g.org/ FOSS4G2009] event, 20-23 October, 2009, in Sydney Australia
 +
|-
 +
|Press Release
 +
|[[FOSS4G_2009_Press_Release_32]]
 +
 +
|}
 +
 
== Participants ==
 
== Participants ==
  
 
* Geoserver
 
* Geoserver
 
** Andrea Aime
 
** Andrea Aime
 +
** [[User:simboss | Simone Giannecchini]]
 
* MapServer
 
* MapServer
 
** [[User:Dmorissette | Daniel Morissette]]
 
** [[User:Dmorissette | Daniel Morissette]]
−
** Jeff McKenna
+
** [[User:jmckenna | Jeff McKenna]]
 
** Paul Ramsey
 
** Paul Ramsey
 +
** Frank Warmerdam (raster)
 
* PostGIS
 
* PostGIS
 
** Paul Ramsey
 
** Paul Ramsey
Line 13: Line 32:
 
* SDE
 
* SDE
 
** Mike Smith
 
** Mike Smith
 +
* ArcGIS Server (withdrew from exercise)
 +
** Satish Sankaran
  
 
== Basic Premise ==
 
== Basic Premise ==
  
−
Each WMS will be set up on the server box to serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to exercise various aspect of those layers.
+
The performance shoot-out presentation at [[FOSS4G2009]] will test how long each Web mapping server takes to generate a map image, from a common set of spatial data, on a common platform.  The data will be served by each Web mapping server through the WMS standard, which will serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to measure various aspects of those layers.
 +
 
 +
== Rules of Engagement ==
 +
 
 +
# All parties must contribute any changes that they make to their software for this exercise, back to their community.
 +
# Comparisons will be made of the best available version of the software, be it a formal release or a development version.  For example, MapServer 5.6.0 (betas) will be tested, while GeoServer 2.0 (dev) will be tested.
 +
# Data formats used must be shapefiles for vectors, geotiff files for rasters, and PostgreSQL/PostGIS for database.  If you wish to test other formats/backends (such as SDE or ECW) you may do so, but that is optional for all parties.  See below for access to the required data.
 +
# Output formats are: jpeg for imagery, gif for non-antialised vector provided we do any, and png24 for antialiased
 +
# Nearest Neighbor resampling should be used for raster image display
 +
 
 +
== Coordination ==
 +
 
 +
Coordination is primarily via the Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking
 +
 
 +
Weekly meetings will occur through IRC chat in the #foss4g channel on irc.freenode.net
 +
 
 +
=== Next IRC meeting ===
 +
 
 +
* Wed Oct 6th: http://timeanddate.com/worldclock/fixedtime.html?year=2009&month=10&day=6&hour=15&min=0&sec=0
 +
** Agenda:
 +
***
 +
 
 +
=== Previous Meetings ===
 +
 
 +
* Wed Sept 23rd: http://timeanddate.com/worldclock/fixedtime.html?year=2009&month=09&day=23&hour=15&min=0&sec=0
 +
** Agenda:
 +
*** moving scripts/files into SVN on BenchmarkingA server
 +
*** LD_LIBRARY_PATH issue on BenchmarkingA server
 +
*** FYI data download available
 +
*** new vector data available on BenchmarkingA server (/opt/data/TIGER-2008)
 +
**** what layers to use
 +
*** what scales should be styled
 +
**** e.g. MapServer scale of [http://labs.gatewaygeomatics.com/cgi-bin/wms_benchmarking_by_merged?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=vector_benchmarking&STYLES=&SRS=EPSG:4269&BBOX=-97.038385,32.545214,-96.516866,32.989369&WIDTH=950&HEIGHT=600&FORMAT=image/png 1:200,000 of Dallas]
 +
**** e.g. MapServer scale of [http://labs.gatewaygeomatics.com/cgi-bin/wms_benchmarking_by_merged?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=vector_benchmarking&STYLES=&SRS=EPSG:4269&BBOX=-97.0286,32.5511,-96.8724,32.6925&WIDTH=950&HEIGHT=600&FORMAT=image/png 1:75,000 of Duncanville/Dallas] (with some ESRI icons)
 +
* Wed Sept 16th: http://timeanddate.com/worldclock/fixedtime.html?year=2009&month=09&day=16&hour=15&min=0&sec=0
 +
** Agenda:
 +
*** Directory structure in /opt/ ?
 +
*** Status of ERDAS
 +
*** status of home for the benchmarking files: http://trac.osgeo.org/osgeo/ticket/435
 +
*** feedback on vector styling
 +
** Minutes
 +
*** msmith: will setup SDE+Oracle on RedHat by Tues
 +
*** augusttown: will draft gnis_pop & tiger_trac styling
 +
*** jmckenna: possibly have maptools.org host the benchmarking files?
 +
*** frankw: press release should for now include GeoServer, MapServer, ESRI, and 'other servers' (until ERDAS confirms)
 +
 
 +
* Wed Sept 9th: http://timeanddate.com/worldclock/fixedtime.html?year=2009&month=09&day=09&hour=15&min=0&sec=0
 +
** Minutes:
 +
*** jmckenna: Need confirmation from ERDAS so press release can move forward
 +
*** msmith:  can setup rhel 5 64 bit with sde 9.3.1 and oracle 11g 64 bit for SDE (need confirmation from ESRI that this setup is OK)
 +
*** aaime: need feedback on styling (icon for point styling, for example)
 +
*** aaime: all should use mailing list more often, for feedback
 +
*** aaime: proposed roads style on mailing list
 +
*** augusttown: are we ok with having a separate machine for database?
 +
*** aaime: postgres should be on that same machine (jmckenna agrees)
 +
*** augusttown: please post possible operating system options for database machine and we will get back to you
 +
*** msmith: pramsey do you want to install postgres/postgis on new db machine?
 +
*** jmckenna: setup an OSGeo server for downloading benchmarking data ([http://trac.osgeo.org/osgeo/ticket/435 ticket])
 +
*** msmith: local port forwarding is ok for doing the testing or do I need to set up the NAT to handle some of the web traffic out?
 +
 
 +
* Wed Sept 2nd: http://timeanddate.com/worldclock/fixedtime.html?year=2009&month=09&day=02&hour=14&min=0&sec=0
 +
** Agenda:
 +
*** update on ERDAS involvement
 +
*** what other web server teams have been contacted?  Should more be contacted?
 +
*** work on a press release: http://lists.osgeo.org/pipermail/benchmarking/2009-August/000107.html
 +
*** output format (jpeg, png 24 bit, png 8 bit, gif?)
 +
*** antialiasing (enabled, or not?)
 +
*** interpolation (nearest neighbour, bilinear, bicubic?)
 +
*** specifics on raster formats tests, especially the TIFF part (multilayers, mosaicked, ...)
 +
 
 +
* Wed Aug 26
 +
** discussion logs: http://logs.qgis.org/foss4g/%23foss4g.2009-08-26.log
 +
** summary:
 +
*** Frank: to confirm with ESRI about Operating System (RHEL or CentOS)
 +
*** Mike: to update wiki with specs when OS switch is done (if required)
 +
*** discussion on how to test (OGC versus native access)...WMS was agreed upon
 +
*** Daniel: volunteers to look at MapServer's WMS vs CGI performance (which is optional)
 +
*** Data: we'll use same data as previous years (but plan to update data to next TIGER or OSM release next year)
 +
**** all agreed to use shapefiles, geotiffs, and PostGIS. to be verified with ESRI.
 +
**** Jeff: to review Andrea's progress re: layer stying
 +
*** foss4g2009: Andrea and Jeff will co-present.  (plus an ESRI representative??)
 +
**** press release needs to be drafted: http://lists.osgeo.org/pipermail/benchmarking/2009-August/000107.html
 +
 
 +
== Hardware ==
 +
 
 +
Machine A (server)
 +
* System Type:  Dell PowerEdge 2850
 +
* Ship Date:    10/21/2004
 +
* 2 X Processor, 80546K,  3.4G, 1M, Xeon Nocona, 800
 +
* 4 X DIMM, 2G 400M, 128X72, 8, 240, 2RX4
 +
* 6 X Hard Drive, 73GB, SCSI, U320, 15K
 +
* Service Tag:    16GRV51
 +
* OS: Centos 5.3 + updates
 +
 
 +
Machine B (testing box)
 +
* System Type:  Dell PowerEdge 1750
 +
* Ship Date:    9/23/2003
 +
* 1 x  Processor, 80532K, 3.06GHz, 512K 533
 +
* 2 x  Dimm, 512, 266M, 64X72, 8K, 184, 1U
 +
* 2 x  Hard Drive, 300Gb, SCSI, U320, 10K
 +
* OS: Centos 5.3 + updates
 +
 
 +
Machine C (DB host)
 +
* Dell Optiplex 755 tower
 +
* 1 x Intel Core2 Duo CPU E6750 @ 2.66GHz / Dual Core
 +
* 100Gb SATA 3.0Gb/s 7200 rpm HD
 +
* 4Gb RAM
 +
* OS: RedHat 5.3 + updates
  
 
== Layers ==
 
== Layers ==
  
−
* '''texas_roads_unstyled'''  This test exercises raw data access speed. The lines are black, the background is white, there are a hundred or so lines in each output image. The output is a two-color PNG8 image. 
+
* '''[[texas_roads_styled]]''' This test exercises the renderer a bit more. The lines will be "pipe-styled" and different road types will have different styles. ''Data: edges_merge''
−
* '''texas_roads_styled''' This test exercises the renderer a bit more. The lines will be "pipe-styled" and different road types will have different styles. TODO(Define the road classes and expected styles.)
+
* '''[[point_layer_styled]]''' This test exercises the ability of the renderer to build a map in which points are symbolized with a certain number of externally provided icons (png/svg) depending on some point attribute. ''Data: gnis_names09''
−
* '''texas_roads_styled_scale''' This test exercises the renderer a bit more. The layer should be a single table/shapefile, the styling would just draw the highways up to a certain scale, and all the roads afterwards. The stress test shall jump between zoom levels to assess the rendering is smart enough to read only the necessary data.
+
* '''[[polygon_layer_styled]]''' Tests polygon rendering, with a solid fill. For this test the type of classification matters less than the number and complexity of the polygons (number of vertices and rings) which is the aspect we should work on. e.g. render maps with a few complex polygons vs other maps with lots of small polygons. ''Data: areawater_merge''
−
* '''texas_roads_styled_labeled''' This test exercises the render and the label algorithms.  Combine the above style with road labels. Labels will follow curves.  TODO(Define the expected labeling look. We know that exact matches between engines will be impossible, this will be a "best efforts" test)
 
−
* '''point_layer_styled''' This test exercises the ability of the renderer to build a map in which points are symbolized with a certain number of externally provided icons (png/svg) depending on some point attribute.  
 
−
* '''polygon layers''' Include one or two test layers to exercise polygon rendering, with solid fill or hatch patterns. For this test the type of classification matters less than the number and complexity of the polygons (number of vertices and rings) which is the aspect we should work on. e.g. render maps with a few complex polygons vs other maps with lots of small polygons.
 
 
* '''single big ECW file''' (actual data TBD). This test exercises the ability to read a single wavelet compressed file and return it in JPEG format.
 
* '''single big ECW file''' (actual data TBD). This test exercises the ability to read a single wavelet compressed file and return it in JPEG format.
 
* '''file system mosaic of TIFF tiles'''. Generated out of the ECW file by splitting it into a sizeable amount of tiles, this test checks the ability of the server to efficiently retrieve data from a large collection of images and deal the associated file handling issues (ulimit and the like). It can also serve as a comparison with ECW results.
 
* '''file system mosaic of TIFF tiles'''. Generated out of the ECW file by splitting it into a sizeable amount of tiles, this test checks the ability of the server to efficiently retrieve data from a large collection of images and deal the associated file handling issues (ulimit and the like). It can also serve as a comparison with ECW results.
Line 31: Line 156:
 
== Data ==
 
== Data ==
  
−
* Data will reside in '''/opt/data''' on the testing server. Do not add any data to this directory without also describing it fully on this page.
+
Data will reside in '''/opt/data''' on the testing server. Do not add any data to this directory without also describing it fully on this page.
 +
 
 +
* Vector data (original):
 +
** '''gnis_names.shp''' (POINT, EPSG:4326) US named feature points.
 +
** '''states.shp''' (POLYGON, EPSG:4326) US states and demographics.
 +
** '''tiger_shp.shp''' (LINESTRING, EPSG:4326) TIGER roads for Texas.
 +
** '''tiger_tracts.shp'''  (POLYGON, EPSG:4326) TIGER census tracts for USA.
 +
* Vector data (GNIS Names 2009)
 +
** '''/GNIS-2009/gnis_names09.shp''' (POINT, EPSG:4269) US named feature points, for 2009.
 +
* Vector data (TIGER 2008 of Texas)
 +
** separated into individual counties (like the TIGER release)
 +
*** '''/TIGER-2008/48_TEXAS/County/arealm.shp''' (POLYGON, EPSG:4269) area landmarks (e.g. parks) for counties.
 +
*** '''/TIGER-2008/48_TEXAS/County/areawater.shp''' (POLYGON, EPSG:4269) area water(e.g. lakes) for counties.
 +
*** '''/TIGER-2008/48_TEXAS/County/edges.shp''' (LINE, EPSG:4269) linework (e.g. roads, rivers) for counties.
 +
*** '''/TIGER-2008/48_TEXAS/County/pointlm.shp''' (POINT, EPSG:4269) point landmarks (e.g. hospital, airport) for counties.
 +
*** '''/TIGER-2008/48_TEXAS/county_tileindex.shp''' (POLYGON, EPSG:4269) county index file used by MapServer.
 +
** merged into single files for the entire state (required for servers that cannot load many .shp files as a single entity)
 +
*** '''/TIGER-2008/48_TEXAS/arealm_merge.shp''' (POLYGON, EPSG:4269) area landmarks (e.g. parks) for entire state.
 +
*** '''/TIGER-2008/48_TEXAS/areawater_merge.shp''' (POLYGON, EPSG:4269) area water(e.g. lakes) for entire state.
 +
*** '''/TIGER-2008/48_TEXAS/edges_merge.shp''' (LINE, EPSG:4269) linework (e.g. roads, rivers) for entire state.
 +
*** '''/TIGER-2008/48_TEXAS/pointlm_merge.shp''' (POINT, EPSG:4269) point landmarks (e.g. hospital, airport) for entire state.
 +
** general
 +
*** '''/TIGER-2008/48_TEXAS/tl_2008_48_place.shp''' (POLYGON, EPSG:4269) places (populated areas) for entire state.
 +
*** '''/TIGER-2008/tl_2008_us_county.shp''' (POLYGON, EPSG:4269) county outlines for entire state.
 +
*** '''/TIGER-2008/tl_2008_us_state.shp''' (POLYGON, EPSG:4269) US state outlines.
 +
* Raster data
 +
** '''world-topo-bathy-200409-3x86400x43200.ecw''' (ECW, EPSG:4326) Medium size ECW file (Bluemarble Next Generation, whole world)
 +
*** '''world-topo-bathy-200409-3x86400x43200.tif''' The same as a tiled BigTIFF, with overviews.
 +
*** '''bluemarble_512/*.tif''' The same split into 512 GeoTIFF tiles, each with overviews.
 +
* PostGIS data
 +
** dbname=benchmark user=postgres password=postgres port=5432 host=localhost
 +
** tables: public.gnis_names public.states public.tiger_shp public.tiger_tracts public.edges_merge public.pointlm_merge public.arealm_merge public.areawater_merge public.gnis_names09
 +
** geometry column: "the_geom" in all cases
 +
** srid: 4326 in all cases
 +
 
 +
=== Download ===
 +
 
 +
* <strike><nowiki>http://www.maptools.org/foss4g/</nowiki> (user/pass: foss4g/foss4g)</strike> (data is no longer available)
 +
**  raster-data.zip (.4 GB)
 +
**  vector-data-original.zip (.5 GB) (old tiger data, merged, for Texas)
 +
**  vector-data-tiger08-tx-counties.zip (1.1 GB) (TIGER 08, stored by county, for Texas)
 +
**  vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
 +
**  GNIS-2009.zip  (7 MB) (2009 GNIS point data for Texas)
 +
 
 +
== Live BenchmarkA WMS GetMap Requests ==
 +
 
 +
=== MapServer ===
 +
 
 +
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged,roads-merged,gnis_names&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png PostGIS]
 +
 
 +
* [http://64.222.187.168/cgi-bin/mapserv560beta3.fcgi?MAP=/opt/benchmarking/mapserver/postgis.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged,roads-merged,gnis_names&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png PostGIS fcgi]
 +
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/shapefile-merged.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged,roads-merged,gnis_names&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png Merged Shapefiles]
 +
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/shapefile-merged.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=roads-merged&STYLES=&SRS=EPSG:4269&WIDTH=950&HEIGHT=600&FORMAT=image/png&BBOX=-96.79344550384396,32.76888304968294,-96.78588564285151,32.77365769873079 Merged Shapefiles - Local Roads labelled]
 +
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/shapefile-county.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-county,roads-county,gnis_names&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png County-based Shapefiles]
 +
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged-oraclespatial,roads-merged-oraclespatial,gnis-names-oraclespatial&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png OracleSpatial]
 +
* [http://64.222.187.168/cgi-bin/mapserv560beta3.fcgi?MAP=/opt/benchmarking/mapserver/oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged-oraclespatial,roads-merged-oraclespatial,gnis-names-oraclespatial&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png OracleSpatial fcgi]
 +
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/oracle.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged-oracle-ogr,roads-merged-oracle-ogr,gnis-names-oracle-ogr&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png Oracle through OGR]
 +
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/sde.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged-sde,roads-merged-sde,gnis-names-sde&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png SDE]
 +
* [http://64.222.187.168/cgi-bin/mapserv560beta3.fcgi?MAP=/opt/benchmarking/mapserver/sde.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged-sde,roads-merged-sde,gnis-names-sde&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png SDE fcgi]
 +
* [http://64.222.187.168/cgi-bin/mapserv560beta3?MAP=/opt/benchmarking/mapserver/sde.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=areawater-merged-sde-ogr,roads-merged-sde-ogr,gnis-names-sde-ogr&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png SDE through OGR]
 +
 
 +
=== GeoServer ===
 +
 
 +
* [http://64.222.187.168:8080/gs20x/ows?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=postgis&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png Postgis group]
 +
* [http://64.222.187.168:8080/gs20x/ows?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=shapefiles&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png Shapefiles group]
 +
* [http://64.222.187.168:8080/gs20x/ows?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=shp:edges_merge&STYLES=roads_classified_shp_label&SRS=EPSG:4269&WIDTH=950&HEIGHT=600&FORMAT=image/png&BBOX=-96.79344550384396,32.76888304968294,-96.78588564285151,32.77365769873079 Shapefiles, labelled]
 +
* [http://64.222.187.168:8080/gs20x/ows?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=oracle&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png Oracle group]
 +
* [http://64.222.187.168:8080/gs20x/ows?SERVICE=WMS&VERSION=1.1.1&REQUEST=getmap&layers=sde&STYLES=&SRS=EPSG:4269&BBOX=-96.980629943844,29.854114257812,-96.79038641357,29.974268066406&WIDTH=950&HEIGHT=600&FORMAT=image/png SDE group]
 +
 
 +
 
  
 
== Software ==
 
== Software ==
Line 44: Line 238:
 
* GDAL 1.6.1 installed in /usr/local (pramsey)
 
* GDAL 1.6.1 installed in /usr/local (pramsey)
 
* AGG 2.5 installed in /usr/local (pramsey)
 
* AGG 2.5 installed in /usr/local (pramsey)
−
* FastCGI (TBD)
+
* FastCGI installed in /usr/local (pramsey)
  
 
=== Databases ===
 
=== Databases ===
Line 52: Line 246:
 
** PostGIS 1.4.0  
 
** PostGIS 1.4.0  
 
** PGDATA in /home/postgres/data
 
** PGDATA in /home/postgres/data
 +
 +
=== GeoServer related ===
 +
* Sun JDK 1.6.0_16 with JAI, JAI Image-IO and ImageIO-ext native extensions installed in (/opt/jdk1.6.0_16) (aaime)
 +
* GeoServer binaries and configurations (/opt/geoserver) (still incomplete at the moment) (aaime)
 +
 +
=== Testing machine ===
 +
 +
* Plain Sun JDK 1.6.0_16 (/opt/jdk1.6.0_16) (aaime)
 +
* JMeter (/opt/jakarta-jmeter-2.3.4) (aaime)
 +
* symbolic link to start jmeter in /usr/local/bin/jmeter (instructions on how to use it will follow) (aaime)
 +
* /usr/local/bin/wms_requests.py, simple tool to generate random bounding boxes and sizes to drive JMeter (aaime, tool developed by Frank W.)
 +
 +
=== Starting / Stopping Services ===
 +
 +
To shutdown apache, and associated mapserver fastcgi processes:
 +
 
 +
  sudo /usr/sbin/apachectl stop
 +
 +
For MapServer team: you might want to do a graceful restart of Apache, before running tests
 +
 +
  sudo /usr/sbin/apachectl graceful
 +
 +
To shutdown arcgis:
 +
 +
  sudo su - arcgis
 +
  cd /arcgis/scripts
 +
  ./stopserver
 +
 +
To shutdown GeoServer do the following
 +
 +
  sudo /opt/geoserver/gs20x/catalina.sh stop
 +
  ps aux | grep java | grep geoserver
 +
 +
The second command can be used to confirm GeoServer is really down, if not, kill it manually ('kill -9 PID').
 +
 +
== SVN ==
 +
 +
The project files (minus data) are stored in Subversion (http://svn.osgeo.org/osgeo/foss4g/benchmarking/).
 +
 +
== Press Release ==
 +
 +
[[FOSS4G_2009_Press_Release_32]]
 +
 +
== Results ==
 +
 +
Slides of the presentation of the results can be 
 +
[http://svn.osgeo.org/osgeo/foss4g/benchmarking/scripts/results/benchmarking2009.odp pulled from the code repository] or can be
 +
[http://www.slideshare.net/gatewaygeomatics.com/wms-performance-shootout viewed online].
 +
 +
Other results obtained during the process can be seen [http://svn.osgeo.org/osgeo/foss4g/benchmarking/scripts/results/ in the svn repository].
 +
 +
== Links ==
 +
 +
* [[FOSS4G_Benchmark | FOSS4G WMS Benchmark]]
 +
* [[Benchmarking_2010 | Benchmarking 2010]]
 +
* [http://sourceforge.net/projects/wmstester/ WMSTester - tool for testing not from OSGeo]
 +
 +
----
 +
 +
[[Category:FOSS4G2009]]
 +
[[Category:Benchmarking]]

Latest revision as of 02:11, 13 February 2015

Overview

Description The performance shoot-out will see mapping servers compared in terms of how long they take to generate a map image, from common set of spatial data.
Participants development teams from GeoServer, MapServer
Results Announced FOSS4G2009 event, 20-23 October, 2009, in Sydney Australia
Press Release FOSS4G_2009_Press_Release_32

Participants

Basic Premise

The performance shoot-out presentation at FOSS4G2009 will test how long each Web mapping server takes to generate a map image, from a common set of spatial data, on a common platform. The data will be served by each Web mapping server through the WMS standard, which will serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to measure various aspects of those layers.

Rules of Engagement

  1. All parties must contribute any changes that they make to their software for this exercise, back to their community.
  2. Comparisons will be made of the best available version of the software, be it a formal release or a development version. For example, MapServer 5.6.0 (betas) will be tested, while GeoServer 2.0 (dev) will be tested.
  3. Data formats used must be shapefiles for vectors, geotiff files for rasters, and PostgreSQL/PostGIS for database. If you wish to test other formats/backends (such as SDE or ECW) you may do so, but that is optional for all parties. See below for access to the required data.
  4. Output formats are: jpeg for imagery, gif for non-antialised vector provided we do any, and png24 for antialiased
  5. Nearest Neighbor resampling should be used for raster image display

Coordination

Coordination is primarily via the Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking

Weekly meetings will occur through IRC chat in the #foss4g channel on irc.freenode.net

Next IRC meeting

Previous Meetings

  • Wed Sept 9th: http://timeanddate.com/worldclock/fixedtime.html?year=2009&month=09&day=09&hour=15&min=0&sec=0
    • Minutes:
      • jmckenna: Need confirmation from ERDAS so press release can move forward
      • msmith: can setup rhel 5 64 bit with sde 9.3.1 and oracle 11g 64 bit for SDE (need confirmation from ESRI that this setup is OK)
      • aaime: need feedback on styling (icon for point styling, for example)
      • aaime: all should use mailing list more often, for feedback
      • aaime: proposed roads style on mailing list
      • augusttown: are we ok with having a separate machine for database?
      • aaime: postgres should be on that same machine (jmckenna agrees)
      • augusttown: please post possible operating system options for database machine and we will get back to you
      • msmith: pramsey do you want to install postgres/postgis on new db machine?
      • jmckenna: setup an OSGeo server for downloading benchmarking data (ticket)
      • msmith: local port forwarding is ok for doing the testing or do I need to set up the NAT to handle some of the web traffic out?
  • Wed Aug 26
    • discussion logs: http://logs.qgis.org/foss4g/%23foss4g.2009-08-26.log
    • summary:
      • Frank: to confirm with ESRI about Operating System (RHEL or CentOS)
      • Mike: to update wiki with specs when OS switch is done (if required)
      • discussion on how to test (OGC versus native access)...WMS was agreed upon
      • Daniel: volunteers to look at MapServer's WMS vs CGI performance (which is optional)
      • Data: we'll use same data as previous years (but plan to update data to next TIGER or OSM release next year)
        • all agreed to use shapefiles, geotiffs, and PostGIS. to be verified with ESRI.
        • Jeff: to review Andrea's progress re: layer stying
      • foss4g2009: Andrea and Jeff will co-present. (plus an ESRI representative??)

Hardware

Machine A (server)

  • System Type: Dell PowerEdge 2850
  • Ship Date: 10/21/2004
  • 2 X Processor, 80546K, 3.4G, 1M, Xeon Nocona, 800
  • 4 X DIMM, 2G 400M, 128X72, 8, 240, 2RX4
  • 6 X Hard Drive, 73GB, SCSI, U320, 15K
  • Service Tag: 16GRV51
  • OS: Centos 5.3 + updates

Machine B (testing box)

  • System Type: Dell PowerEdge 1750
  • Ship Date: 9/23/2003
  • 1 x Processor, 80532K, 3.06GHz, 512K 533
  • 2 x Dimm, 512, 266M, 64X72, 8K, 184, 1U
  • 2 x Hard Drive, 300Gb, SCSI, U320, 10K
  • OS: Centos 5.3 + updates

Machine C (DB host)

  • Dell Optiplex 755 tower
  • 1 x Intel Core2 Duo CPU E6750 @ 2.66GHz / Dual Core
  • 100Gb SATA 3.0Gb/s 7200 rpm HD
  • 4Gb RAM
  • OS: RedHat 5.3 + updates

Layers

  • texas_roads_styled This test exercises the renderer a bit more. The lines will be "pipe-styled" and different road types will have different styles. Data: edges_merge
  • point_layer_styled This test exercises the ability of the renderer to build a map in which points are symbolized with a certain number of externally provided icons (png/svg) depending on some point attribute. Data: gnis_names09
  • polygon_layer_styled Tests polygon rendering, with a solid fill. For this test the type of classification matters less than the number and complexity of the polygons (number of vertices and rings) which is the aspect we should work on. e.g. render maps with a few complex polygons vs other maps with lots of small polygons. Data: areawater_merge
  • single big ECW file (actual data TBD). This test exercises the ability to read a single wavelet compressed file and return it in JPEG format.
  • file system mosaic of TIFF tiles. Generated out of the ECW file by splitting it into a sizeable amount of tiles, this test checks the ability of the server to efficiently retrieve data from a large collection of images and deal the associated file handling issues (ulimit and the like). It can also serve as a comparison with ECW results.

Data

Data will reside in /opt/data on the testing server. Do not add any data to this directory without also describing it fully on this page.

  • Vector data (original):
    • gnis_names.shp (POINT, EPSG:4326) US named feature points.
    • states.shp (POLYGON, EPSG:4326) US states and demographics.
    • tiger_shp.shp (LINESTRING, EPSG:4326) TIGER roads for Texas.
    • tiger_tracts.shp (POLYGON, EPSG:4326) TIGER census tracts for USA.
  • Vector data (GNIS Names 2009)
    • /GNIS-2009/gnis_names09.shp (POINT, EPSG:4269) US named feature points, for 2009.
  • Vector data (TIGER 2008 of Texas)
    • separated into individual counties (like the TIGER release)
      • /TIGER-2008/48_TEXAS/County/arealm.shp (POLYGON, EPSG:4269) area landmarks (e.g. parks) for counties.
      • /TIGER-2008/48_TEXAS/County/areawater.shp (POLYGON, EPSG:4269) area water(e.g. lakes) for counties.
      • /TIGER-2008/48_TEXAS/County/edges.shp (LINE, EPSG:4269) linework (e.g. roads, rivers) for counties.
      • /TIGER-2008/48_TEXAS/County/pointlm.shp (POINT, EPSG:4269) point landmarks (e.g. hospital, airport) for counties.
      • /TIGER-2008/48_TEXAS/county_tileindex.shp (POLYGON, EPSG:4269) county index file used by MapServer.
    • merged into single files for the entire state (required for servers that cannot load many .shp files as a single entity)
      • /TIGER-2008/48_TEXAS/arealm_merge.shp (POLYGON, EPSG:4269) area landmarks (e.g. parks) for entire state.
      • /TIGER-2008/48_TEXAS/areawater_merge.shp (POLYGON, EPSG:4269) area water(e.g. lakes) for entire state.
      • /TIGER-2008/48_TEXAS/edges_merge.shp (LINE, EPSG:4269) linework (e.g. roads, rivers) for entire state.
      • /TIGER-2008/48_TEXAS/pointlm_merge.shp (POINT, EPSG:4269) point landmarks (e.g. hospital, airport) for entire state.
    • general
      • /TIGER-2008/48_TEXAS/tl_2008_48_place.shp (POLYGON, EPSG:4269) places (populated areas) for entire state.
      • /TIGER-2008/tl_2008_us_county.shp (POLYGON, EPSG:4269) county outlines for entire state.
      • /TIGER-2008/tl_2008_us_state.shp (POLYGON, EPSG:4269) US state outlines.
  • Raster data
    • world-topo-bathy-200409-3x86400x43200.ecw (ECW, EPSG:4326) Medium size ECW file (Bluemarble Next Generation, whole world)
      • world-topo-bathy-200409-3x86400x43200.tif The same as a tiled BigTIFF, with overviews.
      • bluemarble_512/*.tif The same split into 512 GeoTIFF tiles, each with overviews.
  • PostGIS data
    • dbname=benchmark user=postgres password=postgres port=5432 host=localhost
    • tables: public.gnis_names public.states public.tiger_shp public.tiger_tracts public.edges_merge public.pointlm_merge public.arealm_merge public.areawater_merge public.gnis_names09
    • geometry column: "the_geom" in all cases
    • srid: 4326 in all cases

Download

  • http://www.maptools.org/foss4g/ (user/pass: foss4g/foss4g) (data is no longer available)
    • raster-data.zip (.4 GB)
    • vector-data-original.zip (.5 GB) (old tiger data, merged, for Texas)
    • vector-data-tiger08-tx-counties.zip (1.1 GB) (TIGER 08, stored by county, for Texas)
    • vector-data-tiger08-tx-merged.zip (1.1 GB) (TIGER 08, merged, for Texas)
    • GNIS-2009.zip (7 MB) (2009 GNIS point data for Texas)

Live BenchmarkA WMS GetMap Requests

MapServer

GeoServer


Software

For those of us who have to build our software, the builds will be staged in /opt/build

Libraries

  • GEOS 3.1.1 installed in /usr/local (pramsey)
  • Proj4 SVN trunk installed in /usr/local (pramsey)
  • libecw installed in /usr/local (pramsey)
  • GDAL 1.6.1 installed in /usr/local (pramsey)
  • AGG 2.5 installed in /usr/local (pramsey)
  • FastCGI installed in /usr/local (pramsey)

Databases

  • PostgreSQL installed in /usr/local/pgsql (pramsey)
    • PostgreSQL 8.3.7
    • PostGIS 1.4.0
    • PGDATA in /home/postgres/data

GeoServer related

  • Sun JDK 1.6.0_16 with JAI, JAI Image-IO and ImageIO-ext native extensions installed in (/opt/jdk1.6.0_16) (aaime)
  • GeoServer binaries and configurations (/opt/geoserver) (still incomplete at the moment) (aaime)

Testing machine

  • Plain Sun JDK 1.6.0_16 (/opt/jdk1.6.0_16) (aaime)
  • JMeter (/opt/jakarta-jmeter-2.3.4) (aaime)
  • symbolic link to start jmeter in /usr/local/bin/jmeter (instructions on how to use it will follow) (aaime)
  • /usr/local/bin/wms_requests.py, simple tool to generate random bounding boxes and sizes to drive JMeter (aaime, tool developed by Frank W.)

Starting / Stopping Services

To shutdown apache, and associated mapserver fastcgi processes: 

 sudo /usr/sbin/apachectl stop

For MapServer team: you might want to do a graceful restart of Apache, before running tests 

 sudo /usr/sbin/apachectl graceful

To shutdown arcgis: 

 sudo su - arcgis
 cd /arcgis/scripts
 ./stopserver

To shutdown GeoServer do the following 

 sudo /opt/geoserver/gs20x/catalina.sh stop
 ps aux | grep java | grep geoserver

The second command can be used to confirm GeoServer is really down, if not, kill it manually ('kill -9 PID').

SVN

The project files (minus data) are stored in Subversion (http://svn.osgeo.org/osgeo/foss4g/benchmarking/).

Press Release

FOSS4G_2009_Press_Release_32

Results

Slides of the presentation of the results can be pulled from the code repository or can be viewed online.

Other results obtained during the process can be seen in the svn repository.

Links

Retrieved from "http://wiki.osgeo.org/w/index.php?title=Benchmarking_2009&oldid=81717"