Difference between revisions of "Benchmarking 2010"
Jump to navigation
Jump to search
Ivansanchez (talk | contribs) |
|||
| Line 96: | Line 96: | ||
{| | {| | ||
| − | | '''Mapping Server''' || '''ext''' || '''type''' | + | | '''Mapping Server''' || '''ext''' || '''type''' || '''command to create''' |
|- | |- | ||
| − | | Mapserver || .qix || quadtree | + | | Mapserver || .qix || quadtree || shptree <shpfile> |
|- | |- | ||
| − | | Mapnik || .index || quadtree | + | | Mapnik || .index || quadtree || shapeindex <shpfile> |
|} | |} | ||
| + | |||
| + | * Note: in your server setup documentation be sure to record the EXACT command used to create the indexes, with appropriate options passed (e.g depth) | ||
== Hardware == | == Hardware == | ||
Revision as of 08:18, 21 July 2010
Basic Premise
Following up on last year's exercise, the performance shoot-out presentation at FOSS4G2010 will test how long each Web mapping server takes to generate a map image, from a common set of spatial data, on a common platform. The data will be served by each Web mapping server through the WMS standard, which will serve exactly the same set of LAYERS. A JMeter load will be run on the testing box to measure various aspects of those layers.
Participants
| Mapping Server | Development Team Leader | IRC nick |
| Cadcorp GeognoSIS | Martin Daly | mpdaly |
| Constellation-SDI | Cédric Briançon | cedricbr |
| Erdas Apollo | Dimitri Monie | dmonie |
| GeoServer | Andrea Aime | aaime |
| MapGuide Open Source | Zac Spitzer | zacspitzer |
| Mapnik | Dane Springmeyer | springmeyer |
| MapServer | Jeff McKenna | jmckenna |
| Oracle MapViewer | Michael Smith | msmith_ |
| QGIS mapserver | Marco Hugentobler | marco___ |
Timeline
| January 1st, 2010 | begin contacting all mapping servers |
| June 1st, 2010 | commitment due by all interested mapping servers |
| June 2nd, 2010 | exercise begins (and weekly meetings) |
| August 1st, 2010 | final testing begins |
| August 23rd, 2010 | no further testing |
| September 6-9, 2010 | present results at FOSS4G2010 |
Rules of Engagement
- All parties must contribute any changes that they make to their software for this exercise, back to their community. Note that the changes don't have to be contributed before the conference, just in a reasonable period of time.
- Comparisons will be made of the best available version of the software, be it a formal release or a development version.
- Two tests will be run: one 'baseline' test with the data in its raw format (with spatial indexes), and another 'best effort' test where 'the sky is the limit' for what changes you want to make to the data (change format, generalize, etc)
- Teams must document all steps they did to manipulate the data/server for both the 'baseline' and 'best effort' tests. If a team does not document the steps on this wiki then that team's test results will not be used.
- Data formats to be used will be shapefiles for vectors, and uncompressed geotiffs for rasters.
- WMS output formats to be used will be png8 and png24 where possible
Datasets
A set of data published by the spanish National Mapping Agency will be used. This data is free for non-commercial use, so it's perfectly OK to use it in the benchmark.
Vector data will be a topographic map, composed of several shapefiles (one shapefile per theme). Raster data will be 50cm/px aerial photography.
We still do not have the definitive datasets, due to their size and the processing time needed to put them together. You can get some sample data by reading Benchmarking 2010/How to get some sample data.
Server Details and Differences
It is the responsibility of each team to document their setup with regard to data. One particular thing that will differ is each team's use of spatial indexes on shapefiles.
Teams are not allowed to modify the .shp, .shx, .dbf or .prj files of the vector data (for the baseline benchmark). They can, however, create auxiliary files for spatial indexes and the like.
| Mapping Server | ext | type | command to create |
| Mapserver | .qix | quadtree | shptree <shpfile> |
| Mapnik | .index | quadtree | shapeindex <shpfile> |
- Note: in your server setup documentation be sure to record the EXACT command used to create the indexes, with appropriate options passed (e.g depth)
Hardware
Machine A (windows server)
- System Type: Dell PowerEdge R410
- Ship Date: 7/7/2010
- Processor: Intel® Xeon® E5630 2.53Ghz, 12M Cache,Turbo, HT, 1066MHz Max Mem
- 8GB Memory (4x2GB), 1333MHz Dual Ranked RDIMMs for 1Processor, Optimized
- 250GB 7.2K RPM SATA, 2TB 7.2K RPM SATA
- OS: Windows Server 64bit
Machine B (linux server)
- System Type: Dell PowerEdge R410
- Ship Date: 7/7/2010
- Processor: Intel® Xeon® E5630 2.53Ghz, 12M Cache,Turbo, HT, 1066MHz Max Mem
- 8GB Memory (4x2GB), 1333MHz Dual Ranked RDIMMs for 1Processor, Optimized
- 250GB 7.2K RPM SATA, 2TB 7.2K RPM SATA
- OS: Centos 5.5 x86-64
Machine C (DB Host)
- System Type: Gateway E6610D
- Processor, Intel Core2 Duo - E6750 2.66 Ghz
- 4Gb Ram
- 250Gb Hard Drive 7200 rpm
- OS: Centos 5.5 x86-64
Machine D (testing server (JMeter))
- System Type: Dell Precision Workstation 390
- Ship Date: 9/7/2006
- Processor, 6300, 1.86, 2M, Core Duo-conroe, Burn 2
- 2Gb RAM
- 160 Gb Hard drive 7200 rpm
- Service Tag: 5Q5LQB1
- OS: Centos 5.5 i386
Communication
Coordination/communication is primarily via the Benchmarking mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking
Weekly meetings will occur through IRC chat in the #foss4g channel on irc.freenode.net
Next IRC meeting
- Wed July 21st @ 15:00:00 UTC
- Agenda:
- update on data processing for all of Spain
- update on server setup
- update on data styling
- checkin on server installs / library dependencies
- Agenda:
Previous IRC meeting
- Wed July 14th @ 15:00:00 UTC
- Agenda:
- update on data processing for all of Spain
- update on server setup
- data styling discussions
- scales to use for layers
- missing full road network
- layer to use for labels along lines
- Summary: (log)
- Attendance:
- Jeff McKenna (jmckenna)
- Andrea Aime (aaime)
- Iván Sánchez (IvanSanchez)
- Michael Smith (msmith_)
- Dimitri Monie (dmonie)
- Dane Springmeyer (springmeyer)
- Discussion:
- data:
- full dataset of Spain, vector and raster, has been processed by IvanSanchez, and he is checking it and will create a torrent for transfer, by the end of the week
- because of shapefile size limit some layers may have to be split (contour-1.shp, contour-2.shp)
- servers:
- msmith has received both server machines
- T1 router is being installed, it will be a dedicated line for this project
- he is adding 2 TB drives to each
- layer styling:
- we notice that the Spain dataset we are using (BCN25) does not include street files (zoomed in road network)
- we agreed to use contours for curved label tests instead
- jmckenna will test the layer scales provided by Marco (QGIS mapserver)
- aaime pointed out problem with the posted SLDs, in terms of the label property (jmckenna to look into it)
- aaime pointed out that we should make sure that all servers compute the scales the same way
- OGC recommends 90 DPI (GeoServer, Mapnik use the recommended 90...MapServer defaults to 72, but should be configurable)
- all teams should look into how their engine handles this and report to group
- other:
- IvanSanchez reported that a Spanish custom-made WMS server may join the exercise (they access raster files only)
- dmonie pointed out that Erdas has been quiet lately because one of the team leaders is on holiday
- data:
- Attendance:
- Agenda:
- Wed June 30th @ 15:00:00 UTC
- Summary: log
- Wed June 30th @ 15:00:00 UTC
- Summary: log
- Wed June 23rd @ 15:00:00 UTC
- Agenda:
- update on proposed layer styling
- update on available server specs
- Summary: (log)
- Agenda:
- Wed June 16th @ 15:00:00 UTC
- Agenda:
- update on server infrastructure
- update on OSGeo-es dataset (Sample data download instructions)
- Summary: (log)
- Attendance:
- Jeff McKenna (jmckenna)
- Andrea Aime (aaime)
- Iván Sánchez (IvanSanchez)
- Michael Smith (msmith_)
- Cédric Briançon (cedricbr)
- Discussion:
- "base" run will use only shapefiles, "best" run will be any format (raster/vector)
- teams can optionally test Oracle, PostGIS, Ingres connections during the "best" run
- layers tested will be: labelled roads (with labels that follow the lines), thematic polygon map, and a test with labelled roads + thematic polygon + point layer in a single request (3 separate runs)
- msmith and jmckenna to examine styling for one sheet (Barcelona, sheet# 420)
- each base and best run will test data in raw projection (ETRS89 + UTM28/29/30/31) and then reprojected on-the-fly to the Google Mercator projection
- threads (number of concurrent requests) will be: 1-2-4-8-16-32-64
- US Army Corps is proposing to provide 4 desktop machines. 2 identical for testing ( win/lin ) and 1 for jmeter and 1 for databases
- msmith will find out availability of 2-cpu machines this week
- Attendance:
- Agenda:
- Wed June 2nd @ 15:00:00 UTC
- Agenda:
- team representative introductions
- Establish this year's 'Rules of Engagement' (previous RoE)
- Discuss dataset to be used (including how to share the large dataset, for running local tests)
- Discuss server infrastructure
- Discuss possible changes to the test script parameters
- Set next meeting date
- Summary: (log)
- Attendance:
- Jeff McKenna, MapServer (jmckenna)
- Cédric Briançon - Constellation (cedricbr)
- Michael Smith - Oracle Mapviewer and Oracle connections in MapServer (msmith_)
- Zac Spitzer - MapGuide (zacspitzer)
- Martin Daly - Cadcorp (mpdaly)
- Anne-Sophie Collignon - ERDAS (ascollignon)
- Marco Hugentobler - QGIS mapserver (marco___)
- Andrea Aime - GeoServer (aaime)
- Dimitri Monie - ERDAS (dmonie)
- Frank Warmerdam - MapServer, possibly Ingres as well (FrankW)
- Iván Sánchez - provide geodata from Spanish sources and help out Mapnik (IvanSanchez)
- Dane Springmeyer - Mapnik (springmeyer)
- Johann Sorel - GeotoolKit / Constellation (Eclesia)
- Adrian Custer - Constellation (acuster)
- Pirmin Kalberer - QGIS mapserver (pirmin_k)
- Trevor Wekel - MapGuide (trevorw)
- Daniel Morissette - MapServer (danmo)
- Discussion:
- All agreed to this year's "Rules of Engagement"
- All parties must contribute any changes that they make to their software for this exercise, back to their community. Note that the changes don't have to be contributed before the conference, just in a reasonable period of time.
- Comparisons will be made of the best available version of the software, be it a formal release or a development version.
- Two tests will be run: one 'baseline' test with the data in its raw format (with spatial indexes), and another 'best effort' test where 'the sky is the limit' for what changes you want to make to the data (change format, generalize, etc)
- Teams must document all steps they did to manipulate the data/server for both the 'baseline' and 'best effort' tests. If a team does not document the steps on this wiki then that team's test results will not be used.
- Data formats to be used will be shapefiles for vectors, and uncompressed geotiffs for rasters.
- WMS output formats to be used will be png8 and png24 where possible
- Dataset
- Iván Sánchez Ortega from OSGeo-es is willing to provide local Spain datasets
- data is for non-commercial use only
- 10 GB of shapefiles
- 100 GB of uncompressed rasters
- Server infrastructure
- possible hosts are msmith (US Army Corps) or Skygone (http://www.skygoneinc.com/)
- need two identical servers, one running Unix and another Windows, plus a client machine
- specs could be: quad core, 8GB of memory, 1TB hard drive
- need fast pipe to server, as large datasets will be transferred
- Next meeting date
- all agreed to use same day and time next week, but limit the meeting length to one hour
- All agreed to this year's "Rules of Engagement"
- Attendance:
- Agenda:
Sample Dataset Styling
- download sample dataset
- layer styling rules