Difference between revisions of "BuildBot Configuration"

From OSGeo
Jump to navigation Jump to search
m (Linked Buildbot page)
 
(31 intermediate revisions by 6 users not shown)
Line 1: Line 1:
 +
See [[Buildbot]] for more information.
 +
 
This document attempts to discuss configuration details of the BuildBot system for OSGeo projects.
 
This document attempts to discuss configuration details of the BuildBot system for OSGeo projects.
  
 
= Operational Configurations =
 
= Operational Configurations =
  
−
{| border=0 cellspacing=5 cellpadding=5 style="text-align: left;"
+
List of configured and running instances of BuildBot is available at OSGeo BuildBot website: http://buildbot.osgeo.org
−
! Project
 
−
! Buildbot URL
 
−
! Status
 
−
|-
 
−
| [http://osgeo.org/gdal_ogr GDAL] || http://buildbot.osgeo.org:8500 || good
 
−
|-
 
−
| [http://fdo.osgeo.org/ FDO] || http://buildbot.osgeo.org:8501 || under construction
 
−
|-
 
−
| [http://proj.maptools.org PROJ.4] || http://buildbot.osgeo.org:8502 || good
 
−
|}
 
  
 
= About BuildBot =
 
= About BuildBot =
Line 19: Line 11:
 
The BuildBot is a system to automate the build and test cycle during a software development process.
 
The BuildBot is a system to automate the build and test cycle during a software development process.
  
−
BuildBot homepage: http://buildbot.sourceforge.net/
+
BuildBot homepage: http://buildbot.net/
  
 
Thanks to Howard Butler's installation of BuildBot, we've been able to test it with some of Open Source GIS projects for a few months.
 
Thanks to Howard Butler's installation of BuildBot, we've been able to test it with some of Open Source GIS projects for a few months.
Line 32: Line 24:
  
 
The '''main objective''' of this initiative is to install and configure BuildBot instances for the OSGeo projects.
 
The '''main objective''' of this initiative is to install and configure BuildBot instances for the OSGeo projects.
−
 
−
==  Hobu's BuildBot instances ==
 
−
 
−
*MapServer - http://mapserver.builds.hobu.net/
 
−
*GDAL (replaced with OSGeo Buildbot) - http://gdal.builds.hobu.net/
 
−
*GEOS - http://geos.builds.hobu.net/
 
  
 
= BuildBot Administration =
 
= BuildBot Administration =
Line 43: Line 29:
 
The prototype infrastructure configuration and testing started by:
 
The prototype infrastructure configuration and testing started by:
  
 +
* [[User:Mloskot|Mateusz Loskot]]
 +
* [[User:Ajolma|Ari Jolma]]
 
* [[User:Hobu|Howard Butler]]
 
* [[User:Hobu|Howard Butler]]
−
* [[User:Mloskot|Mateusz Loskot]]
 
 
* [[User:Frank_Warmerdam|Frank Warmerdam]]
 
* [[User:Frank_Warmerdam|Frank Warmerdam]]
 +
* [[User:Szekerest|Tamas Szekeres]]
  
 
''(add info about administration policies, tasks responsibility, for the OSGeo BuildBot)''
 
''(add info about administration policies, tasks responsibility, for the OSGeo BuildBot)''
Line 52: Line 40:
  
 
Following section describes details of BuildBot infrastructure configuration hosted on one of the OSGeo servers.
 
Following section describes details of BuildBot infrastructure configuration hosted on one of the OSGeo servers.
 +
 +
Current installation of [http://buildbot.net/trac BuildBot] is based on versions:
 +
 +
[buildbot@xblade14 ~]$ buildbot --version
 +
Buildbot version: 0.7.11p3
 +
Twisted version: 8.1.0
 +
 +
== Upgrades ==
 +
 +
* [http://trac.osgeo.org/osgeo/ticket/491 Buildbot 0.7.11p3 upgrade] - steps of upgrade procedure included
 +
 +
== Configuration Files ==
 +
 +
All files of BuildBot configuration are maintained in OSGeo SVN repository (see [http://trac.osgeo.org/osgeo/ticket/102 Ticket #102]) in dedicated module:
 +
 +
https://svn.osgeo.org/osgeo/buildbot/
 +
 +
which is accessible by dedicated SVN user ''buildbot''.
 +
 +
It means, that ''$HOME'' directory of ''buildbot'' user is under revision control, being a working copy of the ''buildbot'' module in SVN repository.
 +
 +
Versioned files do '''NOT''' contain any sensitive data like BuildBot slave passwords. Passwords are being kept out of the versioned tree, dedicated ''etc'' directory, stored in Python script files (.py) in form of regular Python dictionaries. These files are then imported as Python modules where necessary:
 +
 +
import passwd_gdal as secret
 +
 +
The location of password files is added to ''PYTHONPATH''.
 +
 +
All directories and files that should not be stored in the SVN repository have been added to ''svn:ignore'' property. There are no binary files maintained in the repository.
 +
 +
=== Maintenance ===
 +
 +
Every time changes are applied to configuration files, they must be submitted to back to the repository with appropriate log message, for instance:
 +
 +
svn ci -m "buildbot: Added new build step to GDAL slave on telascience." --username buildbot
 +
 +
Please remember, that commit must be issued as SVN user ''buildbot'', but not as any other OSGeo SVN user.
  
 
== Server ==
 
== Server ==
  
−
The OSGeo BuildBot is hosted on one of the [[SAC_Service_Status#Telascience_Blades|Telascience Blades]]:
+
The OSGeo BuildBot Master is hosted on one of the [[SAC_Service_Status#Telascience_Blades|Telascience Blades]]:
  
 
* Hostname: xblade14-2
 
* Hostname: xblade14-2
Line 61: Line 85:
  
 
''(add / link to details about installed development software and versions)''
 
''(add / link to details about installed development software and versions)''
 +
 +
Telascience xblade11-2 (198.202.74.216) is allocated for buildbot slaves.  Some slaves are also setup on xblade14-2 though we are avoiding adding more there if possible.
  
 
== Basic concepts ==
 
== Basic concepts ==
Line 81: Line 107:
 
== BuildBot user ==
 
== BuildBot user ==
  
−
* BuildBot runs as user (also see [[#User_Security|User Security]] section)
+
* BuildBot runs as user (also see [[#User_security|User Security]] section)
  
 
  buildbot
 
  buildbot
Line 87: Line 113:
 
* Home directory for ''buildbot'' user points to:
 
* Home directory for ''buildbot'' user points to:
  
−
  /home/buildbot
+
  /osgeo/buildbot
 +
 
 +
which is also linked to ''/home/buildbot''.
  
 
=== User security ===
 
=== User security ===
Line 223: Line 251:
  
 
*According to this scheme, there is a room for 100 instances of the OSGeo BuildBot (shortly, 100 OSGeo projects).
 
*According to this scheme, there is a room for 100 instances of the OSGeo BuildBot (shortly, 100 OSGeo projects).
 +
 +
= BuildBot Startup =
 +
 +
On xblade14-2 machine, all Buildbot instances are configured to start on boot. This dedicated entry in crontab for ''buildbot'' user on xblade14-2:
 +
 +
$ crontab -l
 +
OSGEOBUILDHOME=/osgeo/buildbot
 +
# On Boot, clean Buildbot logs and start all instances
 +
@reboot $OSGEOBUILDHOME/buildbot_clean_logs.sh
 +
 +
= BuildBot Maintenance =
 +
 +
== Start Buildbot Instance Manually ==
 +
 +
$ cd $OSGEOBUILDHOME
 +
$ ./buildbot_start.sh <INSTANCE DIR>
 +
 +
where <INSTANCE DIR> is name (absolute or relative path) of a Buildbot instance directory. For example, command to start GDAL Buildbot is:
 +
 +
  $ ./buildbot_start.sh gdal
 +
 +
== Stop Buildbot Instance Manually ==
 +
 +
$ cd $OSGEOBUILDHOME
 +
$ ./buildbot_stop.sh <INSTANCE DIR>
 +
 +
where <INSTANCE DIR> is name (absolute or relative path) of a Buildbot instance directory. For example, command to stop PROJ.4 Buildbot is:
 +
 +
  $ ./buildbot_stop.sh proj.4
 +
 +
== Restart Buildbot Instance Manually ==
 +
 +
$ cd $OSGEOBUILDHOME
 +
$ ./buildbot_restart.sh <INSTANCE DIR>
 +
 +
It's also possible to restart all instances at once:
 +
 +
$ login to buildbot.osgeo.org. 
 +
$ sudo su buildbot
 +
$ cd /osgeo/buildbot
 +
$ ./restartall.sh
 +
 +
Often after an unclear reboot, it is necessary to clean the old daemon pid files before a start/restart will work properly.
 +
 +
$ cd /osgeo/buildbot
 +
$ rm */*/twistd.pid
 +
 +
== Logs Cleanup ==
 +
 +
Sometimes, logs and generated files take a lot of disk space. There is a script dedicated to do cleanup for all configured and active Buildbot instances on the xblade14-2 machine. In order clean all logs of all Buildbot instances, run following script as ''buildbot'' user:
 +
 +
$ cd $OSGEOBUILDHOME
 +
$ ./buildbot_clean_logs.sh
  
 
= Additional features proposal =
 
= Additional features proposal =
Line 230: Line 311:
  
 
= OSGeo BuildBot Documentation =
 
= OSGeo BuildBot Documentation =
−
 
−
''(following documents need to be written)''
 
  
 
* All activated [[#BuildBot_instance_per_project|OSGeo BuildBot instances]] are listed here http://buildbot.osgeo.org/
 
* All activated [[#BuildBot_instance_per_project|OSGeo BuildBot instances]] are listed here http://buildbot.osgeo.org/
Line 238: Line 317:
  
 
* [[How to create new OSGeo BuildBot instance]]
 
* [[How to create new OSGeo BuildBot instance]]
−
* [[How to connect new buildslave to an existing buildmaster]]
 
−
* [[How to configure URL redirects for status page of new BuildBot instance]]
 
  
 
= References =
 
= References =
Line 249: Line 326:
 
*[[SAC Service Status]]
 
*[[SAC Service Status]]
 
*[[Project_Infrastructure_Migration#Automated_Build.2FSmoke_Test_System|Automated Build/Smoke Test System]] in the [[Project Infrastructure Migration]]
 
*[[Project_Infrastructure_Migration#Automated_Build.2FSmoke_Test_System|Automated Build/Smoke Test System]] in the [[Project Infrastructure Migration]]
 +
 +
[[Category:Infrastructure]]

Latest revision as of 15:09, 15 October 2011

See Buildbot for more information.

This document attempts to discuss configuration details of the BuildBot system for OSGeo projects.

Operational Configurations

List of configured and running instances of BuildBot is available at OSGeo BuildBot website: http://buildbot.osgeo.org

About BuildBot

The BuildBot is a system to automate the build and test cycle during a software development process.

BuildBot homepage: http://buildbot.net/

Thanks to Howard Butler's installation of BuildBot, we've been able to test it with some of Open Source GIS projects for a few months.

After these tests, we've found that:

  • BuildBot works very well
  • BuildBot has a very positive effect on the development of the projects
  • BuildBot an important communication channel about compilation and testing issues
  • BuildBot helps developers and users to observe a cycle of development in details
  • BuildBot decreases time needed to compile and test changes in a multiplatform environment

The main objective of this initiative is to install and configure BuildBot instances for the OSGeo projects.

BuildBot Administration

The prototype infrastructure configuration and testing started by:

(add info about administration policies, tasks responsibility, for the OSGeo BuildBot)

BuildBot Configuration

Following section describes details of BuildBot infrastructure configuration hosted on one of the OSGeo servers.

Current installation of BuildBot is based on versions: 

[buildbot@xblade14 ~]$ buildbot --version
Buildbot version: 0.7.11p3
Twisted version: 8.1.0

Upgrades

Configuration Files

All files of BuildBot configuration are maintained in OSGeo SVN repository (see Ticket #102) in dedicated module: 

https://svn.osgeo.org/osgeo/buildbot/

which is accessible by dedicated SVN user buildbot.

It means, that $HOME directory of buildbot user is under revision control, being a working copy of the buildbot module in SVN repository.

Versioned files do NOT contain any sensitive data like BuildBot slave passwords. Passwords are being kept out of the versioned tree, dedicated etc directory, stored in Python script files (.py) in form of regular Python dictionaries. These files are then imported as Python modules where necessary: 

import passwd_gdal as secret

The location of password files is added to PYTHONPATH.

All directories and files that should not be stored in the SVN repository have been added to svn:ignore property. There are no binary files maintained in the repository.

Maintenance

Every time changes are applied to configuration files, they must be submitted to back to the repository with appropriate log message, for instance: 

svn ci -m "buildbot: Added new build step to GDAL slave on telascience." --username buildbot

Please remember, that commit must be issued as SVN user buildbot, but not as any other OSGeo SVN user.

Server

The OSGeo BuildBot Master is hosted on one of the Telascience Blades:

  • Hostname: xblade14-2
  • IP: 198.202.74.219

(add / link to details about installed development software and versions)

Telascience xblade11-2 (198.202.74.216) is allocated for buildbot slaves. Some slaves are also setup on xblade14-2 though we are avoiding adding more there if possible.

Basic concepts

The BuildBot system architecture is based on two base concepts:

  • buildmaster - the brain, controls single instance of BuildBot
  • buildslave - the worker controlled by the buildmaster, performs all steps defined for a build

Selected server hosts all instances of the OSGeo BuildBot infrastructure.

A single instance of BuildBot, running on the OSGeo server, consists of single buildmaster and single buildslave.

Development team of particular project is able to connect more buildslaves running on separate machines, remotely. Buildslaves are connected to the buildmaster in a star topology. They connect to the buildmaster over a TCP connection to a publically-visible port (See Buildmaster ports section).

Every project gets its own instance of the OSGeo BuildBot and can connect various external buildslaves.

(add picture showing visualizing the structure of the OSGeo BuildBot infrastructure

BuildBot user

buildbot
  • Home directory for buildbot user points to:
/osgeo/buildbot

which is also linked to /home/buildbot.

User security

buildbot is a normal Linux user with LDAP account.

The access is secured with password.

If you need access to buildbot user privileges, please contact OSGeo BuildBot administrators.

OSGEOBUILDHOME

All BuildBot instances live in common home directory: 

/osgeo/buildbot
  • buildbot user references to BuildBot instances using OSGEOBUILDHOME variable:
OSGEOBUILDHOME=/home/buildbot
  • BuildBot software (Python module) has been installed to:
PYTHONPATH=$OSGEOBUILDHOME/usr/lib/python2.4/site-packages


See BuildBot directories section for details about OSGEOBUILDHOME environment variable.

BuildBot directories

Following subsection explains directories layout used by the OSGeo BuildBot infrastructure.

  • BuildBot infrastructure base directory (later referenced as OSGEOBUILDHOME):
/osgeo/buildbot
  • BuildBot software (Python 2.4 packages) installation prefix:
/osgeo/buildbot/usr

BuildBot web page

(configure HTTP servers and DNS for the BuildBot on the OSGeo servers)

The main URL of the OSGeo BuildBot is 

http://buildbot.osgeo.org

It's dedicated to:

  • list existing BuildBot instances used by OSGeo projects
  • list URLs to BuildBot status pages

NOTE: The BuildBot web page is not yet accessible. Subdomain has not been set up yet, etc.

BuildBot instance per project

  • Base directory for BuildBot instance for one project:
$OSGEOBUILDHOME/<projectname>

where <projectname> is a placeholder for lower-case name of a project.

This path represents directory for single instance of the OSGeo BuildBot used by particular project.

Next, inside a project BuildBot instance directory, we have:

  • buildmaster
$OSGEOBUILDHOME/<projectname>/buildmaster
  • buildslave
$OSGEOBUILDHOME/<projectname>/buildslave
  • The BuildBot status page for particular project can be accessed through:
http://buildbot.osgeo.org/<projectname>/

or using address schema as URL:PORT: 

http://buildbot.osgeo.org:PORT

NOTE: See note about inaccessibility of the BuildBot web page.

GDAL BuildBot example

The OSGeo BuildBot instance for GDAL project (let's call it GDAL BuildBot) is hosted inside the: 

$OSGEOBUILDHOME/gdal

what expands to following path: 

/osgeo/buildbot/gdal

The GDAL BuildBot uses single instance the buildmaster: 

$OSGEOBUILDHOME/gdal/buildmaster

and single instance of the buildslave, local one: 

$OSGEOBUILDHOME/gdal/buildslave

Next, GDAL developers can connect remote buildslaves hosted on various platforms, for exmaple on Mac OS. All buildslaves (one local and N number of remote slaves) are controlled by the GDAL BuildBot buildmaster.

The GDAL BuildBot status page URL is: 

http://buildbot.osgeo.org/gdal/

or 

http://buildbot.osgeo.org:8500

Buildmaster ports

This section explains details of TCP connection ports configuration for buildmaster instances.

Buildmaster of every OSGeo BuildBot instance is configured using two TCP ports:

  • one, to listen on for connections from buildslaves - SLAVEPORT
  • second, used by internal web server to listen on for connection from Web clients requesting for the BUildBot status web page - HTTPPORT

The OSGeo BuildBot infrastructure uses following scheme to assign BuildBot instances with TCP ports:

  • Port number XXYY, where
    • XX identifies port category: SLAVEPORT uses 15 (15YY) and HTTPPORT uses 85 (85YY)
    • YY identifies particular instance of the OSGeo BuildBot, a project
  • SLAVEPORT ports start from 1500 and HTTPPORT ports start from 8500
  • Both ports, SLAVEPORT and HTTPPORT, incement simultaneously, for example:
project A uses 1501 and 8501, where 01 is the instance (project) identifier

project B uses 1527 and 8527, where 27 is the instance (project) identifier
  • According to this scheme, there is a room for 100 instances of the OSGeo BuildBot (shortly, 100 OSGeo projects).

BuildBot Startup

On xblade14-2 machine, all Buildbot instances are configured to start on boot. This dedicated entry in crontab for buildbot user on xblade14-2: 

$ crontab -l
OSGEOBUILDHOME=/osgeo/buildbot 
# On Boot, clean Buildbot logs and start all instances
@reboot $OSGEOBUILDHOME/buildbot_clean_logs.sh

BuildBot Maintenance

Start Buildbot Instance Manually

$ cd $OSGEOBUILDHOME
$ ./buildbot_start.sh <INSTANCE DIR>

where <INSTANCE DIR> is name (absolute or relative path) of a Buildbot instance directory. For example, command to start GDAL Buildbot is: 

 $ ./buildbot_start.sh gdal

Stop Buildbot Instance Manually

$ cd $OSGEOBUILDHOME
$ ./buildbot_stop.sh <INSTANCE DIR>

where <INSTANCE DIR> is name (absolute or relative path) of a Buildbot instance directory. For example, command to stop PROJ.4 Buildbot is: 

 $ ./buildbot_stop.sh proj.4

Restart Buildbot Instance Manually

$ cd $OSGEOBUILDHOME
$ ./buildbot_restart.sh <INSTANCE DIR>

It's also possible to restart all instances at once: 

$ login to buildbot.osgeo.org.  
$ sudo su buildbot
$ cd /osgeo/buildbot
$ ./restartall.sh

Often after an unclear reboot, it is necessary to clean the old daemon pid files before a start/restart will work properly. 

$ cd /osgeo/buildbot
$ rm */*/twistd.pid

Logs Cleanup

Sometimes, logs and generated files take a lot of disk space. There is a script dedicated to do cleanup for all configured and active Buildbot instances on the xblade14-2 machine. In order clean all logs of all Buildbot instances, run following script as buildbot user: 

$ cd $OSGEOBUILDHOME
$ ./buildbot_clean_logs.sh

Additional features proposal

OSGeo BuildBot Documentation

HOWTO

References

List of articles related to / depending on the OSGeo BuildBot installation and configuration project:

Retrieved from "http://wiki.osgeo.org/w/index.php?title=BuildBot_Configuration&oldid=58038"