Benchmarking 2009

Participants

• Geoserver
  • Andrea Aime
• MapServer
  • Daniel Morissette
  • Jeff McKenna
  • Paul Ramsey
• PostGIS
  • Paul Ramsey
• Oracle
  • Mike Smith
• SDE
  • Mike Smith

Basic Premise

Each WMS will be set up on the server box to serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to exercise various aspect of those layers.

Layers

• texas_roads_unstyled This test exercises raw data access speed. The lines are black, the background is white, there are a hundred or so lines in each output image. The output is a two-color PNG8 image.
• texas_roads_styled This test exercises the renderer a bit more. The lines will be "pipe-styled" and different road types will have different styles. TODO(Define the road classes and expected styles.)
• texas_roads_styled_scale This test exercises the renderer a bit more. The layer should be a single table/shapefile, the styling would just draw the highways up to a certain scale, and all the roads afterwards. The stress test shall jump between zoom levels to assess the rendering is smart enough to read only the necessary data.
• texas_roads_styled_labeled This test exercises the render and the label algorithms. Combine the above style with road labels. Labels will follow curves. TODO(Define the expected labeling look. We know that exact matches between engines will be impossible, this will be a "best efforts" test)
• point_layer_styled This test exercises the ability of the renderer to build a map in which points are symbolized with a certain number of externally provided icons (png/svg) depending on some point attribute.
• polygon layers Include one or two test layers to exercise polygon rendering, with solid fill or hatch patterns. For this test the type of classification matters less than the number and complexity of the polygons (number of vertices and rings) which is the aspect we should work on. e.g. render maps with a few complex polygons vs other maps with lots of small polygons.
• single big ECW file (actual data TBD). This test exercises the ability to read a single wavelet compressed file and return it in JPEG format.
• file system mosaic of TIFF tiles. Generated out of the ECW file by splitting it into a sizeable amount of tiles, this test checks the ability of the server to efficiently retrieve data from a large collection of images and deal the associated file handling issues (ulimit and the like). It can also serve as a comparison with ECW results.

Data

• Data will reside in /opt/data on the testing server. Do not add any data to this directory without also describing it fully on this page.

Software

For those of us who have to build our software, the builds will be staged in /opt/build

Libraries

• GEOS 3.1.1 installed in /usr/local (pramsey)
• Proj4 SVN trunk installed in /usr/local (pramsey)
• libecw installed in /usr/local (pramsey)
• GDAL 1.6.1 installed in /usr/local (pramsey)
• AGG 2.5 installed in /usr/local (pramsey)
• FastCGI installed in /usr/local (pramsey)

Databases

• PostgreSQL installed in /usr/local/pgsql (pramsey)
  • PostgreSQL 8.3.7
  • PostGIS 1.4.0
  • PGDATA in /home/postgres/data