Benchmarking 2009

From OSGeo
Revision as of 06:00, 31 August 2009 by Aaime (talk | contribs) (→‎Software)
Jump to navigation Jump to search

Participants

  • Geoserver
    • Andrea Aime
  • MapServer
  • PostGIS
    • Paul Ramsey
  • Oracle
    • Mike Smith
  • SDE
    • Mike Smith
  • ArcGIS Server
    • Satish Sankaran

Basic Premise

Each WMS will be set up on the server box to serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to exercise various aspect of those layers.

Rules of Engagement

  1. All parties must contribute any changes that they make to their software for this exercise, back to their community.
  2. Comparisons will be made of the latest released software version (i.e. "stable") and also development versions of the software (i.e. "open"). For example, MapServer 5.4.2 will be tested as well as MapServer SVN trunk. Similarly GeoServer 1.7.6 will be tested as well as GeoServer SVN trunk.
  3. Data formats used must be shapefiles for vectors, geotiff files for rasters, and PostgreSQL/PostGIS for database. If you wish to test other formats/backends (such as SDE or ECW) you may do so, but that is optional for all parties. See below for access to the required data.

Coordination

Coordination is primarily via the Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking

Weekly meetings will occur through IRC chat in the #foss4g channel on irc.freenode.net

Next IRC meeting

Previous Meetings

  • Wed Aug 26
    • discussion logs: http://logs.qgis.org/foss4g/%23foss4g.2009-08-26.log
    • summary:
      • Frank: to confirm with ESRI about Operating System (RHEL or CentOS)
      • Mike: to update wiki with specs when OS switch is done (if required)
      • discussion on how to test (OGC versus native access)...WMS was agreed upon
      • Daniel: volunteers to look at MapServer's WMS vs CGI performance (which is optional)
      • Data: we'll use same data as previous years (but plan to update data to next TIGER or OSM release next year)
        • all agreed to use shapefiles, geotiffs, and PostGIS. to be verified with ESRI.
        • Jeff: to review Andrea's progress re: layer stying
      • foss4g2009: Andrea and Jeff will co-present. (plus an ESRI representative??)

Hardware

Machine A (server)

  • System Type: Dell PowerEdge 2850
  • Ship Date: 10/21/2004
  • 2 X Processor, 80546K, 3.4G, 1M, Xeon Nocona, 800
  • 4 X DIMM, 2G 400M, 128X72, 8, 240, 2RX4
  • 6 X Hard Drive, 73GB, SCSI, U320, 15K
  • Service Tag: 16GRV51
  • OS: Centos 5.3 + updates

Machine B (testing box)

  • System Type: Dell PowerEdge 1750
  • Ship Date: 9/23/2003
  • 1 x Processor, 80532K, 3.06GHz, 512K 533
  • 2 x Dimm, 512, 266M, 64X72, 8K, 184, 1U
  • 2 x Hard Drive, 300Gb, SCSI, U320, 10K
  • OS: Centos 5.3 + updates

Layers

  • texas_roads_unstyled This test exercises raw data access speed. The lines are black, the background is white, there are a hundred or so lines in each output image. The output is a two-color PNG8 image.
  • texas_roads_styled This test exercises the renderer a bit more. The lines will be "pipe-styled" and different road types will have different styles.
  • texas_roads_styled_scale This test exercises the renderer a bit more. The layer should be a single table/shapefile, the styling would just draw the highways up to a certain scale, and all the roads afterwards. The stress test shall jump between zoom levels to assess the rendering is smart enough to read only the necessary data.
  • texas_roads_styled_labeled This test exercises the render and the label algorithms. Combine the above style with road labels. Labels will follow curves. TODO(Define the expected labeling look. We know that exact matches between engines will be impossible, this will be a "best efforts" test)
  • point_layer_styled This test exercises the ability of the renderer to build a map in which points are symbolized with a certain number of externally provided icons (png/svg) depending on some point attribute.
  • polygon layers Include one or two test layers to exercise polygon rendering, with solid fill or hatch patterns. For this test the type of classification matters less than the number and complexity of the polygons (number of vertices and rings) which is the aspect we should work on. e.g. render maps with a few complex polygons vs other maps with lots of small polygons.
  • single big ECW file (actual data TBD). This test exercises the ability to read a single wavelet compressed file and return it in JPEG format.
  • file system mosaic of TIFF tiles. Generated out of the ECW file by splitting it into a sizeable amount of tiles, this test checks the ability of the server to efficiently retrieve data from a large collection of images and deal the associated file handling issues (ulimit and the like). It can also serve as a comparison with ECW results.

Data

Data will reside in /opt/data on the testing server. Do not add any data to this directory without also describing it fully on this page.

  • Vector data:
    • gnis_names.shp (POINT, EPSG:4326) US named feature points.
    • states.shp (POLYGON, EPSG:4326) US states and demographics.
    • tiger_shp.shp (LINESTRING, EPSG:4326) TIGER roads for Texas.
    • tiger_tracts.shp (POLYGON, EPSG:4326) TIGER census tracts for USA.
  • Raster data
    • world-topo-bathy-200409-3x86400x43200.ecw (ECW, EPSG:4326) Medium size ECW file (Bluemarble Next Generation, whole world)
      • world-topo-bathy-200409-3x86400x43200.tif The same as a tiled BigTIFF, with overviews.
      • world-topo-bathy-*-*.tif The same split into eight non-BigTIFF GeoTIFFs, with overviews.

Software

For those of us who have to build our software, the builds will be staged in /opt/build

Libraries

  • GEOS 3.1.1 installed in /usr/local (pramsey)
  • Proj4 SVN trunk installed in /usr/local (pramsey)
  • libecw installed in /usr/local (pramsey)
  • GDAL 1.6.1 installed in /usr/local (pramsey)
  • AGG 2.5 installed in /usr/local (pramsey)
  • FastCGI installed in /usr/local (pramsey)

Databases

  • PostgreSQL installed in /usr/local/pgsql (pramsey)
    • PostgreSQL 8.3.7
    • PostGIS 1.4.0
    • PGDATA in /home/postgres/data

GeoServer related

  • Sun JDK 1.6.0_16 with JAI, JAI Image-IO and ImageIO-ext native extensions installed in (/opt/jdk1.6.0_16) (aaime)
  • GeoServer binaries and configurations (/opt/geoserver) (still incomplete at the moment) (aaime)

Testing machine

  • Plain Sun JDK 1.6.0_16 (/opt/jdk1.6.0_16) (aaime)
  • JMeter (/opt/jakarta-jmeter-2.3.4) (aaime)
  • symbolic link to start jmeter in /usr/local/bin/jmeter (instructions on how to use it will follow) (aaime)
Retrieved from "http://wiki.osgeo.org/w/index.php?title=Benchmarking_2009&oldid=40641"