FOSS4G Benchmark
FOSS4G WMS Benchmark
Introduction
The FOSS4G WMS Benchmark tests how long each Web mapping server takes to generate a map image, from a common set of spatial data, on a common platform. The data will be served by each Web mapping server through the WMS standard, which will serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to measure various aspects of those layers.
FOSS4G Performance Shoot-out
Following a WMS server performance comparison between Mapserver and GeoServer at FOSS4G 2007 in Victoria, a performance shoot-out presentation at the yearly FOSS4G was institutionalized.
One of the most important 'rules of engagement' for each Web server team is that all parties must contribute any changes that they make to their software for this exercise, back to their community.
- FOSS4G 2010 Barcelona
- FOSS4G 2009 Sydney
- FOSS4G 2008 Cape Town
- FOSS4G 2007 Victoria
Running the FOSS4G WMS Benchmark
The FOSS4G is a useful test suite for your own WMS server installation. This section gives you step-by-step instructions for running the FOSS4G WMS benchmark on different platforms.
SVN
The project files (minus data) are stored in Subversion (http://svn.osgeo.org/osgeo/foss4g/benchmarking/).
Ubuntu 9.10 "Karmic Koala"
Install scripts and data sets
Needed packages:
sudo apt-get install subversion unzip sudo apt-get install jmeter jmeter-http jmeter-junit sudo ln -s /usr/share/jmeter/ /usr/share/jmeter/lib/junit
Get the test scripts and configuration files:
svn checkout http://svn.osgeo.org/osgeo/foss4g/benchmarking foss4g_benchmark
Some scripts need base path /opt/benchmarking
sudo ln -s $(pwd)/foss4g_benchmark /opt/benchmarking
Download the test data sets:
mkdir data cd data #Download data: http://wiki.osgeo.org/wiki/Benchmarking_2009#Download unzip GNIS-2009.zip unzip raster-data.zip unzip vector-data-tiger08-tx-merged.zip unzip vector-data-tiger08-tx-counties.zip
PostgreSQL 8.4
sudo apt-get install postgresql-8.4 postgresql-8.4-postgis
sudo su postgres
psql -c "CREATE ROLE benchmark LOGIN PASSWORD 'benchmark'"
Optional: For local access via socket insert the following line in /etc/postgresql/8.4/main/pg_hba.conf:
local benchmark benchmark trust
createdb -E latin1 --owner benchmark benchmark createlang -d benchmark plpgsql psql -q -d benchmark -f /usr/share/postgresql/8.4/contrib/postgis.sql psql -q -d benchmark -f /usr/share/postgresql/8.4/contrib/spatial_ref_sys.sql psql -q -d benchmark -c "GRANT SELECT, UPDATE, INSERT, DELETE ON TABLE public.geometry_columns TO benchmark;" psql -q -d benchmark -c "GRANT SELECT, UPDATE, INSERT, DELETE ON TABLE public.spatial_ref_sys TO benchmark;"
export PGUSER=benchmark export PGPASSWORD=benchmark
shp2pgsql -I -D GNIS-2009/gnis_names09.shp gnis_names09 | psql -q -h localhost -d benchmark shp2pgsql -I -D TIGER-2008/48_TEXAS/areawater_merge.shp areawater_merge | psql -q -h localhost -d benchmark shp2pgsql -I -D TIGER-2008/48_TEXAS/edges_merge.shp edges_merge | psql -q -h localhost -d benchmark
exit #back to regular user
UMN Mapserver
sudo apt-get install mapserver-bin cgi-mapserver
Apache Fast-CGI:
apt-get install libapache2-mod-fcgid ln -s mapserv /usr/lib/cgi-bin/mapserv.fcgi
/etc/apache2/conf.d/fcgi:
<IfModule fcgid_module>
#fcgi settings (see http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html)
DefaultMaxClassProcessCount 8
</IfModule>
/etc/init.d/apache2 reload
Installation checks
cd foss4g_benchmark/scripts
Basic Mapserver installation check:
shp2img -m ../mapserver/shapefile-merged.map -o /tmp/out.png
Mapserver/Shapefile CGI:
LANG=en_US jmeter -t mapserver/vector/shp_areawater_merge.jmx
Setup connection settings
Set your connection parameters:
WMS_HOST="localhost" PG_CONN="dbname=benchmark user=benchmark password=benchmark port=5432 host=localhost"
Replace connection settings in map files:
sed --in-place -e "s/dbname=benchmark user=postgres password=postgres port=5432 host=192.168.0.8/$PG_CONN/" ../mapserver/postgis.map
Replace WMS host name in JMeter configuration:
FILES=$(find . -name '*.jmx') sed --in-place -e "s/192.168.0.3/$WMS_HOST/" $FILES
Set all jmeter configurations to Mapserver FCGI:
FILES=$(find . -name '*.jmx') sed --in-place -e "s/mapserv560beta3/mapserv/" $FILES sed --in-place -e "s/mapserv560beta3.fcgi/mapserv.fcgi/" $FILES FILES=$(find . -name 'shp_*.jmx') sed --in-place -e "s!cgi-bin/mapserv<!cgi-bin/mapserv.fcgi<!" $FILES
Set LANG when running jmeter:
sed --in-place -e 's!$JMETER_HOME/jmeter!LANG=en_US jmeter!' benchmark
Testing
cd foss4g_benchmark/scripts
Mapserver/Shapefile FCGI:
./benchmark mapserver/vector/shp_areawater_merge.jmx
Mapserver/PostgreSQL FCGI:
./benchmark mapserver/vector/pg_fcgi_areawater_merge.jmx
Run all Mapserver tests:
sh mapservconf | tee mapservconf.log grep -v "Generate" mapservconf.log | grep -v "Waiting"| grep -v "Starting" | grep -v "Created" | grep -v "Tidying" | grep -v "end of run" >mapservconf-summarized.log
Discussion
Please edit the wiki for improvements and add instructions for your platform.
Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking