Difference between revisions of "BuildBot Configuration"
Wiki-Mloskot (talk | contribs) m (Renamed variable from BUILDBOTHOME to OSGEOBUILDBOTHOME) |
Wiki-Mloskot (talk | contribs) (Again, renamed OSGEOBUILDBOTHOME to OSGEOBUILDHOME (shorter)) |
||
| Line 77: | Line 77: | ||
*Environment variables used by ''buildbot'' to run the BuildBot scripts: | *Environment variables used by ''buildbot'' to run the BuildBot scripts: | ||
| − | + | OSGEOBUILDHOME=/home/buildbot | |
| − | PYTHONPATH=$ | + | PYTHONPATH=$OSGEOBUILDHOME/usr/lib/python2.4/site-packages |
| − | See [[#BuildBot_directories|BuildBot directories]] section for details about | + | See [[#BuildBot_directories|BuildBot directories]] section for details about OSGEOBUILDHOME environment variable. |
=== User security === | === User security === | ||
| Line 95: | Line 95: | ||
Following subsection explains directories layout used by the OSGeo BuildBot infrastructure. | Following subsection explains directories layout used by the OSGeo BuildBot infrastructure. | ||
| − | *BuildBot infrastructure base directory (later referenced as | + | *BuildBot infrastructure base directory (later referenced as OSGEOBUILDHOME): |
/osgeo/buildbot | /osgeo/buildbot | ||
| Line 122: | Line 122: | ||
*Base directory for BuildBot instance for one project: | *Base directory for BuildBot instance for one project: | ||
| − | $ | + | $OSGEOBUILDHOME/<projectname> |
where ''<projectname>'' is a placeholder for lower-case name of a project. | where ''<projectname>'' is a placeholder for lower-case name of a project. | ||
| Line 132: | Line 132: | ||
*buildmaster | *buildmaster | ||
| − | $ | + | $OSGEOBUILDHOME/<projectname>/buildmaster |
*buildslave | *buildslave | ||
| − | $ | + | $OSGEOBUILDHOME/<projectname>/buildslave |
*The BuildBot status page for particular project can be accessed through: | *The BuildBot status page for particular project can be accessed through: | ||
| Line 148: | Line 148: | ||
The OSGeo BuildBot instance for GDAL project (let's call it GDAL BuildBot) is hosted inside the: | The OSGeo BuildBot instance for GDAL project (let's call it GDAL BuildBot) is hosted inside the: | ||
| − | $ | + | $OSGEOBUILDHOME/gdal |
what expands to following path: | what expands to following path: | ||
| Line 156: | Line 156: | ||
The GDAL BuildBot uses single instance the buildmaster: | The GDAL BuildBot uses single instance the buildmaster: | ||
| − | $ | + | $OSGEOBUILDHOME/gdal/buildmaster |
and single instance of the buildslave, local one: | and single instance of the buildslave, local one: | ||
| − | $ | + | $OSGEOBUILDHOME/gdal/buildslave |
Next, GDAL developers can connect remote buildslaves hosted on various platforms, for exmaple on Mac OS. All buildslaves (one local and N number of remote slaves) are controlled by the GDAL BuildBot buildmaster. | Next, GDAL developers can connect remote buildslaves hosted on various platforms, for exmaple on Mac OS. All buildslaves (one local and N number of remote slaves) are controlled by the GDAL BuildBot buildmaster. | ||
Revision as of 07:38, 28 September 2006
This document attempts to discuss configuration details of the BuildBot system for OSGeo projects.
About BuildBot
The BuildBot is a system to automate the build and test cycle during a software development process.
BuildBot homepage: http://buildbot.sourceforge.net/
Thanks to Howard Butler's installation of BuildBot, we've been able to test it with some of Open Source GIS projects for a few months.
After these tests, we've found that:
- BuildBot works very well
- BuildBot has a very positive effect on the development of the projects
- BuildBot an important communication channel about compilation and testing issues
- BuildBot helps developers and users to observe a cycle of development in details
- BuildBot decreases time needed to compile and test changes in a multiplatform environment
The main objective of this initiative is to install and configure BuildBot instances for the OSGeo projects.
Hobu's BuildBot instances
- MapServer - http://mapserver.builds.hobu.net/
- GDAL - http://gdal.builds.hobu.net/
- GEOS - http://geos.builds.hobu.net/
BuildBot Administration
The prototype infrastructure configuration and testing started by:
(add info about administration policies, tasks responsibility, for the OSGeo BuildBot)
BuildBot Configuration
Following section describes details of BuildBot infrastructure configuration hosted on one of the OSGeo servers.
Server
The OSGeo BuildBot is hosted on one of the Telascience Blades:
- Hostname: xblade14-2
- IP: 198.202.74.219
(add / link to details about installed development software and versions)
Basic concepts
The BuildBot system architecture is based on two base concepts:
- buildmaster - the brain, controls single instance of BuildBot
- buildslave - the worker, performs all builds tasks
Selected server hosts all instances of the OSGeo BuildBot infrastructure.
A single instance of BuildBot, running on the OSGeo server, consists of single buildmaster and single buildslave.
Development team of particular project is able to connect more buildslaves running on separate machines, remotely. Buildslaves are connected to the buildmaster in a star topology. They connect to the buildmaster over a TCP connection to a publically-visible port (See Buildmaster ports section).
Every project gets its own instance of the OSGeo BuildBot and can connect various external buildslaves.
(add picture showing visualizing the structure of the OSGeo BuildBot infrastructure
BuildBot user
- BuildBot runs as user (also see User Security section)
buildbot
- Home directory for buildbot user points to:
/home/buildbot
- Environment variables used by buildbot to run the BuildBot scripts:
OSGEOBUILDHOME=/home/buildbot
PYTHONPATH=$OSGEOBUILDHOME/usr/lib/python2.4/site-packages
See BuildBot directories section for details about OSGEOBUILDHOME environment variable.
User security
buildbot is a normal Linux user with LDAP account.
The access is secured with password.
If you need access to buildbot user privileges, please contact OSGeo BuildBot administrators.
BuildBot directories
Following subsection explains directories layout used by the OSGeo BuildBot infrastructure.
- BuildBot infrastructure base directory (later referenced as OSGEOBUILDHOME):
/osgeo/buildbot
- BuildBot software (Python 2.4 packages) installation prefix:
/osgeo/buildbot/usr
BuildBot web page
(configure HTTP servers and DNS for the BuildBot on the OSGeo servers)
The main URL of the OSGeo BuildBot is
http://buildbot.osgeo.org
It's dedicated to:
- list existing BuildBot instances used by OSGeo projects
- list URLs to BuildBot status pages
NOTE: The BuildBot web page is not yet accessible. Subdomain has not been set up yet, etc.
BuildBot instance per project
- Base directory for BuildBot instance for one project:
$OSGEOBUILDHOME/<projectname>
where <projectname> is a placeholder for lower-case name of a project.
This path represents directory for single instance of the OSGeo BuildBot used by particular project.
Next, inside a project BuildBot instance directory, we have:
- buildmaster
$OSGEOBUILDHOME/<projectname>/buildmaster
- buildslave
$OSGEOBUILDHOME/<projectname>/buildslave
- The BuildBot status page for particular project can be accessed through:
http://buildbot.osgeo.org/<projectname>/
NOTE: See note about inaccessibility of the BuildBot web page.
GDAL BuildBot example
The OSGeo BuildBot instance for GDAL project (let's call it GDAL BuildBot) is hosted inside the:
$OSGEOBUILDHOME/gdal
what expands to following path:
/osgeo/buildbot/gdal
The GDAL BuildBot uses single instance the buildmaster:
$OSGEOBUILDHOME/gdal/buildmaster
and single instance of the buildslave, local one:
$OSGEOBUILDHOME/gdal/buildslave
Next, GDAL developers can connect remote buildslaves hosted on various platforms, for exmaple on Mac OS. All buildslaves (one local and N number of remote slaves) are controlled by the GDAL BuildBot buildmaster.
The GDAL BuildBot status page URL is:
http://buildbot.osgeo.org/gdal/
Buildmaster ports
This section explains details of TCP connection ports configuration for buildmaster instances.
Buildmaster of every OSGeo BuildBot instance is configured using two TCP ports:
- one, to listen on for connections from buildslaves - SLAVEPORT
- second, used by internal web server to listen on for connection from Web clients requesting for the BUildBot status web page - HTTPPORT
The OSGeo BuildBot infrastructure uses following scheme to assign BuildBot instances with TCP ports:
- Port number XXYY, where
- XX identifies port category: SLAVEPORT uses 15 (15YY) and HTTPPORT uses 85 (85YY)
- YY identifies particular instance of the OSGeo BuildBot, a project
- SLAVEPORT ports start from 1500 and HTTPPORT ports start from 8500
- Both ports, SLAVEPORT and HTTPPORT, incement simultaneously, for example:
project A uses 1501 and 8501, where 01 is the instance (project) identifier
project B uses 1527 and 8527, where 27 is the instance (project) identifier
- According to this scheme, there is a room for 100 instances of the OSGeo BuildBot (shortly, 100 OSGeo projects).
OSGeo BuildBot Documentation
(following documents need to be written)
- All activated OSGeo BuildBot instances are listed here http://buildbot.osgeo.org/
HOWTO
- How to create new OSGeo BuildBot instance ?
- How to configure buildmaster ?
- How to connect new buildslave to an existing buildmaster ?
- How to configure URL redirects for status page of new BuildBot instance ?
References
List of articles related to / depending on the OSGeo BuildBot installation and configuration project: