Benchmarking 2010/Constellation-SDI - Revision history

Wiki-Acuster: Add DBF fix

2010-09-08T13:42:12Z

Add DBF fix

Wiki-Acuster: Added the first part of the Saturday testing session.

2010-09-04T12:26:17Z

Added the first part of the Saturday testing session.

Wiki-Acuster: Add a session on performance for tiny datasets.

2010-09-04T12:03:21Z

Add a session on performance for tiny datasets.

Wiki-Acuster: Add a second, clean pass on the rasters

2010-09-03T20:10:41Z

Add a second, clean pass on the rasters

Wiki-Acuster: Add second vector runs

2010-09-03T19:29:53Z

Add second vector runs

Wiki-Acuster: clarify runs; clarify passes

2010-09-03T16:52:46Z

clarify runs; clarify passes

Wiki-Acuster: Add second runs for raster

2010-09-03T16:46:36Z

Add second runs for raster

Wiki-Cedricbr: /* Geotoolkit */

2010-09-03T14:31:55Z

Geotoolkit

Wiki-Acuster: Drop duplicate entry

2010-09-03T13:25:18Z

Drop duplicate entry

Wiki-Acuster: Add some analaysis

2010-09-03T13:13:18Z

Add some analaysis

@@ Line 35: / Line 35: @@
 * Shapefile: reduce styling to one single pass when painting by symbol rather than by feature.
 * Shapefile: reduce the reading of non-necessary parts of the files.
 * DataSource: Enable startup from a coverage mosaic, either a folder or a manager.

@@ Line 187: / Line 187: @@
 === Session 2010.09.04 ===
-This session set out to look at performance not limited by the IO bottleneck.
+This session was split into two parts.
-''Note that this is merely an academic exercise since for any non-trivial dataset, the size of data on disk will be so much larger than the size of available memory that these numbers will never be achieved.'''
+The first part of the session aimed to look at the difference in performance when running with the correct configuration (which had mistakenly not been used in yesterday's session).
 In order to generate this behaviour, the session involved a bunch of runs with the same requests as with the earlier session but with the scripts cut up and restructured so that the data would already have been read by the time the requests are made. The approach yielded somewhat stable numbers with a variability between runs of around 10%.
@@ Line 217: / Line 247: @@
 |}
 === Session NEXT ===

← Older revision		Revision as of 12:03, 4 September 2010
Line 183:		Line 183:

	Not a dramatic change although these numbers are much less consistent than those generated with the older scripts. We can even burst up to well over 60 images per second for raster and 10 images per second for vector in some passes showing the danger of playing on this cusp for any meaningful results.		Not a dramatic change although these numbers are much less consistent than those generated with the older scripts. We can even burst up to well over 60 images per second for raster and 10 images per second for vector in some passes showing the danger of playing on this cusp for any meaningful results.
		+
		+
		+	=== Session 2010.09.04 ===
		+
		+	This session set out to look at performance not limited by the IO bottleneck.
		+
		+	''Note that this is merely an academic exercise since for any non-trivial dataset, the size of data on disk will be so much larger than the size of available memory that these numbers will never be achieved.'''
		+
		+	In order to generate this behaviour, the session involved a bunch of runs with the same requests as with the earlier session but with the scripts cut up and restructured so that the data would already have been read by the time the requests are made. The approach yielded somewhat stable numbers with a variability between runs of around 10%.
		+
		+	The results reported are the last run performed for any thread group. The runs with 32 threads and 64 threads were done with three single passes to save time and to ensure the requests would not outstrip main memory.
		+
		+	{\| border="1" style="text-align:center"
		+	! colspan ="9" \| Raster 3rd Pass
		+	\|-
		+	! rowspan ="2" \| Threads !! colspan="4" \| 25831 !! colspan="4" \| 3857
		+	\|-
		+	! Results !! ----- !! ----- !! ----- !! Results !! ----- !! ----- !! -----
		+	\|-
		+	\| 1 \|\| 13.2 \|\| --- \|\| --- \|\| --- \|\| 11.6 \|\| --- \|\| --- \|\| ---
		+	\|-
		+	\| 2 \|\| 21.6 \|\| --- \|\| --- \|\| --- \|\| 19.6 \|\| --- \|\| --- \|\| ---
		+	\|-
		+	\| 4 \|\| 32.8 \|\| --- \|\| --- \|\| --- \|\| 40.1 \|\| --- \|\| --- \|\| ---
		+	\|-
		+	\| 8 \|\| 34.8 \|\| --- \|\| --- \|\| --- \|\| 30.1 \|\| --- \|\| --- \|\| ---
		+	\|-
		+	\| 16 \|\| 34.9 \|\| --- \|\| --- \|\| --- \|\| 34.1 \|\| --- \|\| --- \|\| ---
		+	\|-
		+	\| 32 \|\| 35.7 \|\| --- \|\| --- \|\| --- \|\| 33.7 \|\| --- \|\| --- \|\| ---
		+	\|-
		+	\| 64 \|\| 35.3 \|\| --- \|\| --- \|\| --- \|\| 32.4 \|\| --- \|\| --- \|\| ---
		+	\|}
		+

@@ Line 65: / Line 65: @@
 '''Note:''' All values reported are in units of responses per second taken from the "Throughput" column of the 'summarizer.py' script. The values are those of the third pass, after the jmeter scripts have looped through the two warmup passes for the various thread counts, from one to sixty-four; we report only the measure from the last pass.
@@ Line 124: / Line 126: @@
 The processing power (8 CPUs) of the machine does not seem to have been stressed at any point in the test runs.
-The numbers differ in numbers generated on local servers which had a twice as big separation between vector and raster results. Variability also seems high enough that several runs would be needed to discriminate between the various configurations, enough so that we probably need to tighten up the testing to get anything meaningful from numbers such as these.
+The numbers differ from the results obtained on local servers which had a more distinct separation between vector and raster results. Variability also seems high enough that several runs would be needed to discriminate between the various configurations, enough so that we probably need to tighten up the testing to get anything meaningful from numbers such as these.
@@ Line 130: / Line 132: @@
 === Session 2010.09.03 afternoon (older, repeating scripts) ===
-Repeating the earlier work, with the older scripts which leave the files in memory.
+Repeating the earlier work, with the older scripts which may leave the file blocks in memory.
 {| border="1" style="text-align:center"
@@ Line 140: / Line 143: @@
 !             Run 1  !!  Run 2  !!  Run 3  !!  Run 4  !!  Run 1  !!  Run 2  !!  Run 3  !!  Run 4
 |-
-|    1    ||   5.5   ||   ---   ||   ---   ||   ---   ||   5.9   ||   ---   ||   ---   ||   ---
+|    1    ||   5.5   ||  11.6   ||   ---   ||   ---   ||   5.9   ||   5.9   ||   ---   ||   ---
 |-
-|    2    ||   4.0   ||   ---   ||   ---   ||   ---   ||   6.9   ||   ---   ||   ---   ||   ---
+|    2    ||   4.0   ||  11.9   ||   ---   ||   ---   ||   6.9   ||   6.7   ||   ---   ||   ---
 |-
-|    4    ||   5.3   ||   ---   ||   ---   ||   ---   ||   5.4   ||   ---   ||   ---   ||   ---
+|    4    ||   5.3   ||   5.5   ||   ---   ||   ---   ||   5.4   ||   5.5   ||   ---   ||   ---
 |-
-|    8    ||   5.1   ||   ---   ||   ---   ||   ---   ||   5.3   ||   ---   ||   ---   ||   ---
+|    8    ||   5.1   ||   5.1   ||   ---   ||   ---   ||   5.3   ||   5.2   ||   ---   ||   ---
 |-
-|   16    ||   5.3   ||   ---   ||   ---   ||   ---   ||   5.3   ||   ---   ||   ---   ||   ---
+|   16    ||   5.3   ||   5.3   ||   ---   ||   ---   ||   5.3   ||   5.1   ||   ---   ||   ---
 |-
-|   32    ||   5.3   ||   ---   ||   ---   ||   ---   ||   5.2   ||   ---   ||   ---   ||   ---
+|   32    ||   5.3   ||   5.2   ||   ---   ||   ---   ||   5.2   ||   5.1   ||   ---   ||   ---
 |-
-|   64    ||   5.3   ||   ---   ||   ---   ||   ---   ||   5.0   ||   ---   ||   ---   ||   ---
+|   64    ||   5.3   ||   5.0   ||   ---   ||   ---   ||   5.0   ||   5.1   ||   ---   ||   ---
 |}
@@ Line 179: / Line 182: @@

@@ Line 163: / Line 163: @@
 !             Run 1  !!  Run 2  !!  Run 3  !!  Run 4  !!  Run 1  !!  Run 2  !!  Run 3  !!  Run 4
 |-
-|    1    ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---
+|    1    ||   1.9   ||   ---   ||   ---   ||   ---   ||   1.8   ||   ---   ||   ---   ||   ---
 |-
-|    2    ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---
+|    2    ||   2.2   ||   ---   ||   ---   ||   ---   ||   2.1   ||   ---   ||   ---   ||   ---
 |-
-|    4    ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---
+|    4    ||   2.6   ||   ---   ||   ---   ||   ---   ||   2.7   ||   ---   ||   ---   ||   ---
 |-
-|    8    ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---
+|    8    ||   2.4   ||   ---   ||   ---   ||   ---   ||   2.5   ||   ---   ||   ---   ||   ---
 |-
-|   16    ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---
+|   16    ||   2.5   ||   ---   ||   ---   ||   ---   ||   2.6   ||   ---   ||   ---   ||   ---
 |-
-|   32    ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---
+|   32    ||   2.3   ||   ---   ||   ---   ||   ---   ||   2.4   ||   ---   ||   ---   ||   ---
 |-
-|   64    ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---   ||   ---
+|   64    ||   2.0   ||   ---   ||   ---   ||   ---   ||   2.0   ||   ---   ||   ---   ||   ---
 |}

@@ Line 64: / Line 64: @@
 This section details the results of running the jmeter scripts against the Constellation-SDI WMS server.
-'''Note:''' All values reported are in units of responses per second taken from the "Throughput" column of the 'summarizer.py' script. The values are those of the third pass, after the jmeter scripts have looped through the two warmup passes for the various thread counts, from one to sixty-four, and then we measure the last pass.
+'''Note:''' All values reported are in units of responses per second taken from the "Throughput" column of the 'summarizer.py' script. The values are those of the third pass, after the jmeter scripts have looped through the two warmup passes for the various thread counts, from one to sixty-four; we report only the measure from the last pass.
-=== Session 2010.09.03 ===
+=== Session 2010.09.03 (New, non-repeating scripts) ===
 This was the first full run of Constellation-SDI using the jmeter scripts.
@@ Line 77: / Line 77: @@
 {| border="1" style="text-align:center"
-! colspan ="9" | Raster Results for 2010.09.03 (New, non-repeating scripts)
+! colspan ="9" | Raster 3rd Pass (New, non-repeating scripts)
 |-
 ! rowspan ="2" | Threads !! colspan="4" | 25831  !! colspan="4" | 3857
@@ Line 100: / Line 100: @@
 {| border="1" style="text-align:center"
-! colspan ="9" | Vector Results for 2010.09.03
+! colspan ="9" | Vector 3rd Pass (New, non-repeating scripts)
 |-
 ! rowspan ="2" | Threads !! colspan="4" | 4326  !! colspan="4" | 3857
@@ Line 134: / Line 134: @@
 {| border="1" style="text-align:center"
-! colspan ="9" | Raster Results
+! colspan ="9" | Raster 3rd Pass (older, repeating scripts)
 |-
 ! rowspan ="2" | Threads !! colspan="4" | 25831  !! colspan="4" | 3857
@@ Line 157: / Line 157: @@
 {| border="1" style="text-align:center"
-! colspan ="9" | Vector Results for 2010.09.03
+! colspan ="9" | Vector 3rd Pass (older, repeating scripts)
 |-
 ! rowspan ="2" | Threads !! colspan="4" | 4326  !! colspan="4" | 3857
@@ Line 186: / Line 186: @@
 {| border="1" style="text-align:center"
-! colspan ="9" | Raster Results
+! colspan ="9" | Raster 3rd Pass
 |-
 ! rowspan ="2" | Threads !! colspan="4" | 25831  !! colspan="4" | 3857
@@ Line 209: / Line 209: @@
 {| border="1" style="text-align:center"
-! colspan ="9" | Vector Results for 2010.09.03
+! colspan ="9" | Vector 3rd Pass
 |-
 ! rowspan ="2" | Threads !! colspan="4" | 4326  !! colspan="4" | 3857

@@ Line 41: / Line 41: @@
 * Renderer: improve decimation algorithm for vector layers.
 * Renderer: switch to OGC conformant 96dpi assumption rather than industry standard 72.
-* Renderer: fix sld parsing errors to handle <litteral> or its absence.
+* Renderer: fix sld parsing errors to handle <ogc:Literal> or its absence.
 * Renderer: optimize the colour model selected for multiple inputs.

← Older revision		Revision as of 13:25, 3 September 2010
Line 37:		Line 37:

	* DataSource: Enable startup from a coverage mosaic, either a folder or a manager.		* DataSource: Enable startup from a coverage mosaic, either a folder or a manager.
−	* DataSource: Create a GeoTiff reader.

	* Renderer: bypass rendering engine for single raster requests.		* Renderer: bypass rendering engine for single raster requests.

← Older revision		Revision as of 13:13, 3 September 2010
Line 125:		Line 125:
	The processing power (8 CPUs) of the machine does not seem to have been stressed at any point in the test runs.		The processing power (8 CPUs) of the machine does not seem to have been stressed at any point in the test runs.

−		+	The numbers differ in numbers generated on local servers which had a twice as big separation between vector and raster results. Variability also seems high enough that several runs would be needed to discriminate between the various configurations, enough so that we probably need to tighten up the testing to get anything meaningful from numbers such as these.