Live GIS Build

From OSGeo
Revision as of 03:20, 20 January 2013 by Kalxas (talk | contribs)
Jump to navigation Jump to search

Getting started

The first thing you will have to do is check-out the latest build scripts using Subversion (SVN): 

svn checkout https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk
  • see here for more details.

How to add your project to OSGeo Live

Creating a fresh Virtual Machine

Refer to: Live_GIS_Virtual_Machine.

Virtual Machine specific steps

Take a copy of the Virtual Machine, as you don't want VMWare tools installed on the ISO image (in the next step).

Install VMWare tools

VMWare tools allow cut and paste from the host operating system to a VMWare virtual machine, and also speeds up graphic rendering on VMWare.

In the VMWare browser control panel, select the virtual machine, then "Summary" tab. Turn on the virtual machine, then select "Install VMWare tools". This mounts the vmware-tools scripts in the arramagong virtual machine, so that it can be installed. Which you can do using the following commands: 

osgeolive$ cd /tmp
osgeolive$ tar -zxf /media/cdrom0/VMwareTools-7.7.5-156745.tar.gz
osgeolive$ cd vmware-tools-distrib/
osgeolive$ sudo ./vmware-install.pl
osgeolive$ # answer defaults to all questions

Power off

Then poweroff the virtual machine. (Note that this removes all files in the tmp/ directory). 

arramagong$ sudo halt

Package the VMWare VM

Note: Most of this step is covered in the package.sh script. Check and update the variables in the script before running on an Ubuntu based system.

On the host computer, all temporally files can be removed from the virtual machine folder: log files, ram files, etc. Remove all files EXCEPT: *.vmx (VM definition file) and *.vmdk (virtual disk file)

Shrink the virtual machine: (requires VMWare Server installed) Note: If you get an error from this command, you may need to temporarily rename the *.vmx file before operation and then name it back after 

host$ vmware-vdiskmanager -k *.vmdk

Zip the image up: 

 host$ 7z a -mx=9 ArramagongGISVM2009alpha4.7z ArramagongGISVM2009alpha4/

If the image is greater than 2 Gig, then you also need to split the image. The OSGeo download server isn't configured to accept files of a greater size. 

host$ split -b 1500M ArramagongGISVM2009alpha4.7z ArramagongGISVM2009alpha4.7z

Create the md5sum checksums, so which can be used to confirm that the images have been downloaded correctly: 

host$ md5sum *.7z*

Build the Live DVD ISO image

Build ISO

This section describes the new method for building OSGeoLive as described in official ubuntu wiki. This section is self-contained and there is no need to perform any of the procedures described above.

All you need is a running Ubuntu/Xubuntu/Kubuntu installation (even within a virtual machine as long as it has ~20GB free disk space). All needed to be done are the following steps under a "user" account:

  • Bootstrap the host operating system. If you use the system to build more than once, then this must be done only for the first build
host$ cd /tmp
host$ wget https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/bootstrap.sh
host$ chmod a+x bootstrap.sh
host$ sudo ./bootstrap.sh

This will install subversion, and the install scripts, and create a link to them from your home directory.

  • Set the Version Number and Changes
 Update https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/VERSION.txt with the current version number.

 Update https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/CHANGES.txt with changes since the last release.

This list can be a summary of the revision log between releases Commit the changes to svn right before a release build.

  • Execute the build script:
host$ cd ~/gisvm/bin
host$ svn up
host$ sudo ./build_chroot.sh 2>&1 | tee /var/log/osgeolive/chroot-build.log
  • Compress the logs:
host$ cd ~/livecdtmp
host$ tar czf version-log.tar.gz -C /var/log osgeolive
  • After the completion of the above script the new iso file is located in ~/livecdtmp along with the build logs. In case you wish to rerun the build process, do not remove or move the xubuntu official iso located in this folder to skip downloading it again.
  • It is required to reboot your host machine after build is completed
  • You can now use the iso to simply install in a virtual machine in order to get the OSGeoLive VM version. The installation procedure is the same as a normal Ubuntu OS.

How to do development / debugging with the current build method

We have created a debug build process so that projects can now easily create their own iso, including parts of the OSGeoLive (eg only one project) in order to test if the installer scripts work well under this new build method. Here are the steps to debug/test your application:

One time steps

You will need to create a pure Xubuntu Virtual Machine setup:

  • Download xubuntu-12.04-desktop-i386.iso from xubuntu web site.
  • Download and install VirtualBox.
  • Create a fresh VM installation of Xubuntu. You will need to create a virtual disk drive with at least 25 GB of space and allocate 768MB of RAM to the VM. During installation set the username to "user" and hostname to "osgeolive". DO NOT install system updates during xubuntu installation or after the installation is done. At this momment we work with the default kernel included in xubuntu.
  • After the VM is done, login as "user" and open a terminal.
  • Bootstrap the VM:
osgeolive$ cd /tmp
osgeolive$ wget https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/bootstrap.sh
osgeolive$ chmod a+x bootstrap.sh
osgeolive$ sudo ./bootstrap.sh

This will install subversion, and the install scripts, and create a link to them from your home directory.

Steps to create the build (repeat as much as needed)

  • Make changes to your project's installation script and commit to svn.
  • Update the subversion code:
osgeolive$ cd ~/gisvm
osgeolive$ svn up
  • Open file inchroot_debug.sh with an editor and comment out all scripts you do not need for your test.
osgeolive$ cd ~/gisvm/bin
osgeolive$ vim inchroot_debug.sh
  • Always leave un-commented the following scripts: setup.sh, install_services.sh, install_mysql.sh, install_java.sh, install_apache2.sh, install_tomcat6.sh, install_desktop.sh and setdown.sh. We have already commented out all project installation scripts (except the needed ones) for you.
  • Save your changes and execute the build:
osgeolive$ cd ~/gisvm/bin
osgeolive$ sudo ./build_chroot_debug.sh 2>&1 | tee /var/log/osgeolive/chroot-build.log
  • After a while the iso will be created in ~livecdtmp/
  • Do not delete the file ~livecdtmp/xubuntu-12.04-desktop-i386.iso as it will be needed for next build (saves time not to download again)
  • Logs are created at /var/log/osgeolive/chroot-build.log
  • Copy the iso and test

Previous Method To Build ISO

The ISO will be built as part of the main.sh script and stored in /tmp/remastersys/ . Make sure to copy the ISO onto the osgeo server, and then remove /tmp/remastersys/ or reboot the VM to remove it(must boot up again) before packaging the VM.

  • Run the script to build the ISO without additional files.
sudo ./build_iso.sh mini 2>&1 | tee /var/log/osgeolive/build_iso-mini.log
  • Run the script to build the ISO, this includes downloading close to 1GB of Windows and Mac installers.
sudo ./build_iso.sh 2>&1 | tee /var/log/osgeolive/build_iso.log
  • Once the ISO is complete copy it out to a server (a local server is fastest)
scp /tmp/remastersys/osgeolive-livedvd-2.0-final.iso.md5 user@server.org:destination/path/
scp /tmp/remastersys/osgeolive-livedvd-2.0-final.iso user@server.org:destination/path/
  • And/OR Wget or scp the file to the upload.osgeo.org server (Note wget is much faster if you have a good webserver to host from)

Package the VM

Note: Most of this step is covered in the package.sh script. Check and update the variables in the script before running on an Ubuntu based system.

On the host computer, all temporally files can be removed from the virtual machine folder: log files, ram files, etc. Remove all files EXCEPT: *.vmx (VM definition file) and *.vmdk (virtual disk file)

Shrink the virtual machine: (requires VMWare Server installed) Note: If you get an error from this command, you may need to temporarily rename the *.vmx file before operation and then name it back after 

host$ vmware-vdiskmanager -k *.vmdk

Zip the image up: 

 host$ 7z a -mx=9 ArramagongGISVM2009alpha4.7z ArramagongGISVM2009alpha4/

If the image is greater than 2 Gig, then you also need to split the image. The OSGeo download server isn't configured to accept files of a greater size. 

host$ split -b 1500M ArramagongGISVM2009alpha4.7z ArramagongGISVM2009alpha4.7z

Create the md5sum checksums, so which can be used to confirm that the images have been downloaded correctly: 

host$ md5sum *.7z*

Backup files

Optional: To save time and bandwidth, it is a good idea to back up all the files downloaded into your /tmp directory and the apt-get cache (var/cache/apt), which you can then copy onto future images you create before starting the build process. These files are copied into /tmp/<version>/ during the build process: 

host$ scp -pr user@<arramagong ip>/tmp/3.1alpha1 arragmagong_tmp_3.1alpha1

Upload the Release

Upload to sourceforge

As of 6.0 the official releases are hosted on sourceforge. To upload you need a sourceforge account and permissions to the osgeo-live project upload. 

rsync -e ssh osgeo-live-5.5.iso username,osgeo-live@frs.sourceforge.net:/home/pfs/project/o/os/osgeo-live/5.5/

Upload to the OSGeo Server

host$ scp -pr osgeolive-gisvm-2.0-alpha5 username@upload.osgeo.org:/osgeo/download/livedvd/

Update the index.html file at: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/download/index.html

Check the result at: http://download.osgeo.org/livedvd

Creating a torrent file

Ibiblio has offered torrent hosting, this section needs to include who to contact/how to setup Several tools exist for creating torrents. Below are the key settings you need.

Deluge with the torrent creator plugin has been tested.

  • Make sure you have a copy of the file you want to create a torrent for. Mounted remote drives may work.
  • Web Seed - URL to a web server that hosts the file, adds speed to the torrents especially at the start. (Most clients implement this now)
  • Tracker - There are 2 open and free trackers that have been tested, use one or the other as most clients do not handle multiple trackers yet.
http://tracker.openbittorrent.com/announce
udp://tracker.openbittorrent.com:80/announce

OR 

http://tracker.publicbt.com:80/announce
udp://tracker.publicbt.com:80/announce
  • Upload the .torrent file of your iso file for others to grab
  • Start the torrent on your machine or dedicated seeder, so that others have somewhere to start.
  • There is a possibility of seeding from osgeo machines but a Quality of Service (QoS) system would need to be in place to ensure it does not impact other services.

SVN branches and tags

Naming

  • main development happens in "trunk" (aka HEAD)
  • a branch is split off before release time. Bug fixes (only) happen in it and updates like version 2.1 are tagged from it.
its name might be like "arramagong_2"
  • a tag is a snapshot of the svn at some point in time, aka a release name. It does not make sense to checkin fixes to one. They should ideally be set as read-only after creation to prevent this.
its name might be like "release_20090927_arramagong_2_0"

Creating

cd livedvd/gisvm/

svn copy trunk branches/arramagong_2
svn commit branches/arramagong_2 -m "Splitting off branch for 2.x"

cd branches/arramagong_2/bin/
# something like this, maybe need a for loop or xargs:
sed -e 's+gisvm/trunk+gisvm/branches/arramagong_2+g' *.sh | less   # check
sed -i -e 's+gisvm/trunk+gisvm/branches/arramagong_2+g' *.sh       # execute
cd -
 
svn copy branches/arramagong_2 tags/release_20090927_arramagong_2_0_3
svn commit tags/release_20090927_arramagong_2_0_3 -m "tag release 2.0.3"

Merging

To merge a change from trunk into a release branch use "svn merge", as follows: (in this example r2131 from trunk) 

svn up branches/arramagong_2/
cd branches/arramagong_2/

svn merge -c 2131 https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk
svn diff
svn commit -m "bugfix: bikeshed should be orange! (merge from trunk r2131)"

If you do this a lot you might make a little shell script. Here's one called svn_merge_livefromtrunk.sh: 

#!/bin/sh
# Usage: svn_merge_livefromtrunk <rev number>
svn merge -c $1 https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk

See also

Retrieved from "http://wiki.osgeo.org/w/index.php?title=Live_GIS_Build&oldid=68456"