Difference between revisions of "Benchmarking 2010/Constellation-SDI"
Wiki-Acuster (talk | contribs) (Initial layout of the page.) |
Wiki-Acuster (talk | contribs) (Expand the benchmark design section) |
||
| Line 1: | Line 1: | ||
This page describes the experience of the Constellation-SDI team during the FOSS4G 2010 Benchmarking effort. | This page describes the experience of the Constellation-SDI team during the FOSS4G 2010 Benchmarking effort. | ||
| − | == | + | == Benchmark design == |
| − | We expended a great deal of effort attempting to understand how a benchmarking effort could be designed properly. The 2009 and 2010 efforts were undertaken in the naive belief that one simply sets up different servers on the same data and makes requests of the servers to compare response times. | + | We expended a great deal of effort attempting to understand how a benchmarking effort could be designed properly. |
| + | |||
| + | The 2009 and 2010 efforts were undertaken in the naive belief that one simply sets up different servers on the same data and makes requests of the servers to compare response times. Belatedly, the 2010 effort is demonstrating that a proper benchmark is more complex, since the data might be in a format useful for a certain class of use cases but meaningless for another scale of usage, the testing can easily be hardware bound limiting useful comparison between servers, servers can be set up to be doing very different work especially in 'best effort'/anything goes configurations, and results can be compared only in superficial ways or in very narrow types of requests. In order to tackle these issues rather than merely pretend they were not serious, we examined what it would take to develop a useful benchmarking protocol, either to stress all the functionality of one particular server or to compare the performance and abilities of various, arbitrary WMS servers. | ||
| + | |||
| + | Developing a WMS benchmarking design which provides useful, comparative metrics of server performance is exceedingly hard. In a recent presentation at the Java Language Summit, Joshua Bloch presented a talk entitled [http://wiki.jvmlangsummit.com/images/1/1d/PerformanceAnxiety2010.pdf ''Performance Anxiety''] which describes the impossibility of developing performant software from first principles in any language due to the enhancements of compilation and machine instruction re-ordering, the necessity of testing to obtain concrete results, and the difficulty of developing proper, statistically rigourous testing metrics. | ||
| + | |||
| + | Since these issues were apparent to us even before this effort, we have been developing tools, benchmark designs and analytic methodologies to test the Constellation-SDI server. This work has been greatly extended during the FOSS4G 2010 benchmarking effort and expanded to consider how to test different WMS servers, possibly built for different uses. | ||
== Enhancements == | == Enhancements == | ||
| − | This section describes enhancements due to the work during the benchmarking 2010 effort, including improved understanding and workflow by the Constellation-SDI team and ameliorations to the code bases of the [http://www.geotoolkit.org/ Geotoolkit.org] library and to the server itself. | + | This section describes enhancements due to the work during the benchmarking 2010 effort, including improved understanding and workflow by the Constellation-SDI team and ameliorations to the code bases of the [http://www.geotoolkit.org/ Geotoolkit.org] library and to the [http://www.constellation-sdi.org Constellation-SDI] server itself. |
=== Benchmarking === | === Benchmarking === | ||
| Line 14: | Line 20: | ||
=== Geotoolkit === | === Geotoolkit === | ||
| − | |||
Revision as of 01:29, 31 August 2010
This page describes the experience of the Constellation-SDI team during the FOSS4G 2010 Benchmarking effort.
Benchmark design
We expended a great deal of effort attempting to understand how a benchmarking effort could be designed properly.
The 2009 and 2010 efforts were undertaken in the naive belief that one simply sets up different servers on the same data and makes requests of the servers to compare response times. Belatedly, the 2010 effort is demonstrating that a proper benchmark is more complex, since the data might be in a format useful for a certain class of use cases but meaningless for another scale of usage, the testing can easily be hardware bound limiting useful comparison between servers, servers can be set up to be doing very different work especially in 'best effort'/anything goes configurations, and results can be compared only in superficial ways or in very narrow types of requests. In order to tackle these issues rather than merely pretend they were not serious, we examined what it would take to develop a useful benchmarking protocol, either to stress all the functionality of one particular server or to compare the performance and abilities of various, arbitrary WMS servers.
Developing a WMS benchmarking design which provides useful, comparative metrics of server performance is exceedingly hard. In a recent presentation at the Java Language Summit, Joshua Bloch presented a talk entitled Performance Anxiety which describes the impossibility of developing performant software from first principles in any language due to the enhancements of compilation and machine instruction re-ordering, the necessity of testing to obtain concrete results, and the difficulty of developing proper, statistically rigourous testing metrics.
Since these issues were apparent to us even before this effort, we have been developing tools, benchmark designs and analytic methodologies to test the Constellation-SDI server. This work has been greatly extended during the FOSS4G 2010 benchmarking effort and expanded to consider how to test different WMS servers, possibly built for different uses.
Enhancements
This section describes enhancements due to the work during the benchmarking 2010 effort, including improved understanding and workflow by the Constellation-SDI team and ameliorations to the code bases of the Geotoolkit.org library and to the Constellation-SDI server itself.
Benchmarking
Geotoolkit
Constellation-SDI
Performance Results
This section details our understanding of the performance of the server.