Wiki-JulienSam: JulienSam good and bad ideas

2013-09-22T14:19:25Z

JulienSam good and bad ideas

Arnulf at 13:31, 22 September 2013

2013-09-22T13:31:34Z

Arnulf: Created page with "Arnulf's notes for the closing plenary talk at FOSS4G 2013

 5 minutes  The Shootout that did not happen.  The Benchmark ... that did not happen. The WPS comparison:  Ho..."

2013-09-22T13:24:10Z

Created page with "Arnulf's notes for the closing plenary talk at FOSS4G 2013 <pre> 5 minutes The Shootout that did not happen. The Benchmark ... that did not happen. The WPS comparison: Ho..."

New page

Arnulf's notes for the closing plenary talk at [[FOSS4G 2013]]

<pre>
5 minutes

The Shootout that did not happen.
The Benchmark ... that did not happen.
The WPS comparison:
How can we do more?
How can we do better?
We need dedication. Throughout the year. Involve research and academia.
Involve sponsors.
Make a committee?
<pre>

Please discuss, add ideas, etc.:
...

← Older revision		Revision as of 14:19, 22 September 2013
Line 15:		Line 15:

	Please discuss, add ideas, etc.:		Please discuss, add ideas, etc.:
−	...	+
		+
		+	A few points to discuss (in no particular order):
		+
		+	The goods:
		+	* There's great interest for a benchmark from the general population, just because it's interesting to see projects have a friendly fight or simply because it is somewhat informative.
		+	* From the project developpers point of view, it's really helpful and all developpers involved could see huge benefits out of it.
		+	* From a marketing point of view, the winning teams (those at the top of the chart) have one more tool to sell their project.
		+
		+	The bads
		+	* This exercise has become (ie always been) very heavy for the developpers involved. In my sense, the 2010 was such a success because the people didn't knew in what they were getting into. I don't think we can ask the propretary vendor to get back for a similar exercise.
		+	* Project champions all told us that they were burnt out at the end. This is not good.
		+	* The scope (speed) may be to narrow as a yearly subject to keep being interesting to all party involved. (Not sure of this one)
		+	* The benefits of having a FOSS4G oriented exercise is not ''that'' appealing to projects. Do projects want more from this time and energy investment than a cool presentation with charts.
		+
		+	The potential solutions:
		+	* The time required to integrate the benchmark exercise should be reduce for this to work. We can't ask Jeff or Andrea (To only mention those two) to spend 4 months (or more) of their (unfunded) time on this.
		+	* We need sponsors for both hardware and money.
		+	* We need a Sprint to get all project champions together and get this out of the door in a week
		+	* This could/should a service offered to participating project where they can run the benchmark and see the results. This would help to see how projects evolve performance wise over time. Thinking about it, this would make a really interesting presentation.
		+	* The test case needs to be kept at minimal. Start small. Render 10 million points with no style for example. I would see between one and three tests the first year so the steps to join are not an issue.
		+	* We may have to remove the hardcore tuning optimisation from the benchmark if that's what is taking time.
		+
		+	Other ideas (Replace those that you don't agree with):
		+	* It would be cool to have a simple testsuite that anyone could run on any computer and see the performance of their project. Just like I have testsuite to test my video card, I would be able to test my MapServer installation. Maybe the benchmark could become only that...
		+	* Our potential sponsors are not projects, but institution or company using the software (Ordnance Survey for example). This is the people we have to aim when planning what the exercise should become. What are their benefits out of this? Is it to be able to chose the right software for the right job? Maybe we could allow sponsors to define their usecase and add that to the test suite.

Talk:Benchmarking 2013 - Revision history

Wiki-JulienSam: JulienSam good and bad ideas

Arnulf at 13:31, 22 September 2013

Arnulf: Created page with "Arnulf's notes for the closing plenary talk at FOSS4G 2013
5 minutes The Shootout that did not happen. The Benchmark ... that did not happen. The WPS comparison: Ho..."

@@ Line 12: / Line 12: @@
 Involve sponsors.
 Make a committee?
-<pre>
+</pre>
 Please discuss, add ideas, etc.:
 ...