Live GIS Build

From OSGeo
Jump to navigation Jump to search

About

Arramagong is an Xubuntu based Virtual Machine and Live DVD which has been installed with a suite of the best Open Source Geospatial software.

How to add your project or data to the LiveGIS / Arramagong

All that's required to add your favorite package into the build is to:

  • Recommended: Subscribe to the Live Demo email list to discuss issues.
If you are unfamiliar with subversion then you can send the script to the OSGeo live-demo mailing list as an attachment and we'll see that it makes it in.
  • Notify the LiveGIS team to test your script, and will ensure to reference your script from main.sh.

Example scripts

You can see example scripts at: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/

The script may be as simple as: install_mapserver.sh, which just "apt-get install mapserver"

Projects that haven't been packaged yet are slightly more complicated:

The Udig package is a good example: install_udig.sh

  • 3rd Party Repositories: Create an <app>.list file with your repository in sources.list.d/ in the svn, in your script copy the file to the local machine like
wget https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/sources.list.d/ubuntugis.list \
  --output-document=/etc/apt/sources.list.d/ubuntugis.list

don't forget to add the repository key like so 

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 68436DDF

Packaging conventions

  • sudo: Assume the script is to be executed as root.
This means that ~/ refers to the directory /home/root/, so you should use ~user/ to reference /home/user/.
Preferred method is to put the USER and USER_HOME variables at the top, so these can be easily changed in all scripts if needed.
USER=user
USER_HOME=/home/${user}
dosomething $USER_HOME
  • Called twice: Assume that the script may be executed twice, (as people re-run the install process to get it right).
Make sure that nothing is corrupted if the script runs a second time.
  • No Prompts: Try not to have any steps which prompt the user for an answer, as we want the scripts to be run automatically. In particular, try running the script a second time, and see if you get prompted to overwrite files.
  • /tmp: Copy all downloads into /tmp/build_<package>
Don't delete the /tmp/build_<package> directory. If the script is rerun, then time and bandwidth is saved by not having to download again.
  • Icons: Where appropriate, create an icon on the desktop which loads the application.
see install_qgis.sh for an example
Create an icon in the the OSGeo Applications pull down list. see install_qgis.sh for an example
  • Browser bookmarks: If the application is launched from the browser, then include the application in Firefox bookmarks.
(Todo: How do we do this?)
  • wget: When downloading large files using wget, use the "-c" continue command, as it will not re-download files if they already exist, and will continue if one is partially downloaded:
wget -c package.tar.gz
  • mkdir: Use -p (parents) option when creating a directory, as it doesn't complain about creating the directory a second time.
mkdir -p <dir>

Writing docs

Definition doc

Each project is to have a distinct sentence or two defining the project, as a html snippet inside a <li>tag</li>. There is a template here: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/doc/template_definition.html . The debian tile should provide a suitable description for .deb packaged projects. Provided by: 

dpkg -s $PACKAGE | grep '^Description:' | cut -f2- -d' '

Save the file here: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/doc/descriptions/project_definition.html

Description Doc

Each project is to also have up to one page description about how to use the project on the live DVD, written as a .odt format (which can be edited with Open Office, or AbiWord. See example here: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/doc/template_description.odt

Once complete, save your docs into <project>_description.odt, and email the Live Demo list to ensure it is reviewed by the community.

You may include links to locally installed pdf or html manuals.

Save the file here: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/doc/descriptions/project_definition.odt

Licence Doc

Each project should create one table row for each licence offered by the project. A template for the file to be produced is here: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/doc/template_licence.html

Save the file here: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/doc/descriptions/project_licence.html

List the project

Once your application is added in SVN please make sure the information about it is up to date on the status table at: http://spreadsheets.google.com/ccc?key=0Al9zh8DjmU_RdGIzd0VLLTBpQVJuNVlHMlBWSDhKLXc&hl=en_GB

Build Drivers

Which version?

The criteria used to select applications for the LiveGIS are as follows:

  1. Priority goes to Ubuntu packaged software first, then Debian packaged software (but often those are the same thing). This facilitates easy and reliable maintenance and update. Users will benefit from it. Programmers are encouraged to move their software into DebianGIS or UbuntuGIS repositories.
  2. Stable, always! LiveGIS users are mainly starters. So they obviously are better with stable software. They have enough problems already and will gladly be happier without bleeding-edge software bugs.

For people who want the latest version, they can create a modified upgrade script.

Low Memory

Both Virtual Machines, and a LiveDVD images are likely to be constrained by limited memory. So to reduce memory usage. Disk image size is not of major concern, as we can just distribute less data.

The following principles should be followed.

  1. Do not start applications upon power up. (Ie, don't start deamons, allow users to start them instead).
  2. Set up examples which, by default, don't depend on other applications. Less applications open, means less memory. Ie, Have GeoServer access a shapefile instead of PostGIS.
  3. Try to avoid scenarios which write data to disk, as disk space in the Live DVD is stored in RAM, and is not cleared afterward.

Directory Structure

  1. /tmp Temporary files (e.g., downloaded archives) go into /tmp. Please create a separate folder for your project.
  2. /usr/lib application are usually installed into /usr/lib
  3. /usr/bin things that get executed by the user such as startup scripts or links to them should go into /usr/bin
  4. /etc/init.d startup/shutdown scripts for services (e.g., postgres, apache, tomcat) are stored in /etc/init.d
  5. /usr/local/share sample data and documentation goes into /usr/local/share
  6. /etc config files are stored in /etc
  7. /home/user user specific config files or working directories can go into /home/user/. However, keep files in /home/user as small as possible as this folder is loaded into memory in the Live DVD. Symlinks into /usr/local can be useful here.


  • You are strongly encouraged to use /usr/local/ and /var/local/ for storing non-packaged content.

Mounting a Virtual Image

For the purposes of faster dissemination of updates to the Live image via something like rsync you can mount an image file. Make sure you turn off the virtual machine before you attempt to mount it, and that you unmount it before you attempt to run the vm again.

Mount VMWare Server

This method assumes you have vmware server installed, the key is that the vmware-mount script is on your system somewhere. This instructions use /space/virtual as the vmware installation folder and /space/virtual/machines as the location of the disk images. For more information see the VMware mounting guide

  • If it complains about not finding libdir/lib/libcrypto.so.0.9.8/libcrypto.so.0.9.8
  • It's because it's looking for it in your vmware bin/libdir which doesn't exist so symlink
sudo ln -s /space/virtual/lib/vmware/ libdir
  • Temporarily move your .vmx file out of the directory
  • Move to you vmware bin folder
cd /space/virtual/bin/
sudo ./vmware-mount -p /space/virtual/machines/gisvm20090828x.vmdk
  • Should be partition 1
  • Make a directory to mount to somewhere easy to find, in my case where I keep my vm images
mkdir /space/virtual/machines/mountedimage
  • Mount it
sudo ./vmware-mount /space/virtual/machines/gisvm20090828x.vmdk 1 /space/virtual/machines/mountedimage
  • Don't forget to umount when you're done
sudo ./vmware-mount -d /space/virtual/machines/mountedimage

Mount VirtualBox

These instructions have not been tested yet, they require VirtualBox 2.0+ http://forums.virtualbox.org/viewtopic.php?f=7&t=17574

Mount the ISO as a readonly filesystem

Not recommeded, it's preferred to get th image right to start, you will not be able to edit files inside the squashfs this way.
 mkdir test_iso
 mount -t iso9660 -o loop,ro livedvd.iso ./test_iso

If you want to do this without su rights, use the FuseISO package. 

fuseiso -p livedvd.iso ./test_iso.$$

Unmount the FuseISO filesystem with: 

fusermount -u ./test_iso.$$

Mount the ISO for changes

If you're on ubuntu to can edit and remaster and ISO with ISOMaster 

sudo apt-get install isomaster

On Windows try Daemon tools and related software.

Creating a fresh Virtual Machine

System Requirements

  • RAM: You can only just get away with 1 Gig. 2 Gig is what you will require to avoid slowdowns due to memory swapping.
  • About 10 Gig of spare hard disk for building a Virtual Machine.(20GB if you intend to build an ISO as well.)
  • Recommended: An Ubuntu based linux distribution. We built this Live DVD process using an Ubuntu based system, and some of the steps assume the hosting operating system is Ubuntu, however you should be able to achieve all the steps (some slightly modified) from Windows.

Selecting a Virtual Machine

The arramagong Live DVD can be built using a range of virtual machines. Here we explain our preferences and reasons:

VMware Server
VMware server provides a browser based interface to the virtual machine which can be accessed over the internet. (This is a good way to let others view your virtual machine without them having to download the large Virtual Machine)

VMware Server also provides compression tools which we use at the end of the build process. (These are not provided by VMWare Player) Unfortunately, there are instability issues with VMWare 2.0.2 + Ubuntu 9.04 / 9.10 (as a host), which means we have been unable to use this combination to build Arramagong 3.0. (The compression tools still seem to work). The vmware images that are created can be used with VMWare Server, VMWare Player and relatively easily can be converted to VirtualBox format. (We haven't worked out how to convert from VirtualBox back to VMware images yet).

VMware Player
VMware Player provides a desktop application which runs a Virtual Machine, and can be used to create a vmware image. It doesn't provide the image compression utilities. (Can we download these from somewhere?)
VirtualBox
VirtualBox is provided as Open Source and can be downloaded onto an Ubuntu based system using apt-get which is great. Unfortunately, we haven't worked out how to convert from VirtualBox back to VMware images yet, and so far VMware Player software is more widely deployed than VirtualBox. So for the moment, we plan to build vmware images then convert to virtual box (rather than the other way around).
Click2Try
http:/click2try.com provide a web site for trialing virtual machines. They built an arramagong 2.0 release for FOSS4G 2009, and as of Feb 2009, have been working on setting an environment set up for 3.0. This should provide a good development environment once set up, but we will need to have a parallel VM development as well as I don't think they will not be building a VM for us.

Install VMWare Server

Note: During the building of Arramagong 3.0, we have found VMWare Server installed on Ubuntu 9.04 and 9.10 to be unstable to the point of being unusable.

We have successfully installed used a CentOS 5.3 host + VMware Server 2.0.2 (Make sure you don't accept any security updates as this introduces the bug which makes VMWare unstable). We suspect that Ubuntu 8.04 (LTS) + VMware should work as it is supposed to be supported, but we haven't verified this yet.

Alternatively, you can build images in VMWare Player.

You will need VMWare server which provides all the tools for configuring a VM, setting the disk size, compressing the image etc. VirtualBox doesn't seem to have the compression scripts required, and it doesn't seem to have a way to copy your image to VMWare.

If you are installing on linux, the installer will ask a series of questions. Answer the default for everything, except the default administrator. For this, you need to enter in your <user_id> for the local computer you are installing on.

Once running, you should be able to access vmware server at: 

https://127.0.0.1:8333

As explained here, firefox has issues connecting to VM until you fix it by doing the following:

  • Go to the url: about:config
  • Update the following:
    • security.enable.ssl2 from false to true

Create an Xubuntu Virtual Machine

If you already have an Xubuntu (or similar) Virtual Machine, you can skip this step.

Otherwise, run VMWare Server, and create an image.

Download the latest Xubuntu ISO. Copy to the Virtual Machine directory: 

wget http://mirror.internode.on.net/pub/ubuntu/xubuntu/9.04/release/xubuntu-9.04-desktop-i386.iso
mv xubuntu-9.04-desktop-i386.iso "/var/lib/vmware/Virtual Machines/"

Run vmware, which opens up a browser based User Interface. 

vmware

Create a new Virtual Machine with:

  • Virtual Machine name = "arramagong"
  • RAM = 512 Gig (we might want to experiment with less)
  • Hard Disk = 20 Gig (If you plan to make an ISO you will need the space, just the VM install will fit in 8GB with little extra room, but it is a pain to increase later, and can't be done from VMWare Player)
  • Mount the XUbuntu ISO image for the disk drive

Run the Virtual Machine, which will boot from the Xubuntu CD. Select to install Xubuntu, using defaults, plus:

What is your name?
user
What is your password?
user
What is the name of your computer?
arramagong
[x] Log in automatically

After installing xubuntu, in the VMWare Summary tab, select the change the CD drive from pointing to the xubuntu iso to point to the actual hard drive.

Alternative: build ISO using vmbuilder

Side note: vmbuilder is reported to be a good way to create an Ubuntu image. I haven't tried it, and haven't worked out if it can create the smaller Xubuntu image. (Please update this wiki if you try it). Note: This method requires you to be running the same or newer version of Ubuntu than you want to make an image for. 

host$ sudo apt-get install python-vm-builder
host$ sudo vmbuilder vmserver ubuntu --suite karmic --arch i386 --mem 512 --rootsize=20000 --user user --pass user

Backup base Virtual Machine

You should now have a XUbuntu Virtual Machine image in /var/lib/vmware/Virtual Machines/ . It would be a good idea to save a copy of this machine somewhere, so you can use it again.

Creating a Click2Try Virtual Machine

Click2Try provides a public website that people can use to create a Virtual Machine, and our project has been given a free account to use Click2Try as a development platform, which makes collaborative development much easier.

As of 20 Feb 2010, the click2try website were having technical difficulties getting an Xunbutu 9.10 base image working. They are still working on it.

Click2Try Account details

Go to http://www.click2try.com and login with:

  • User Name: osgeo
  • password: .ask Cameron Shorter.

Once logged in, click on the “My Dashboard” link located on the upper right corner of the page. Then click the “Application Manager” tab. You will notice that there are two virtual machines already defined in your dashboard. They are:

base-xubuntu-9-04
This is a base xubuntu virtual machine. Do not alter this one. Create copies from this VM when you want a fresh virtual machine to start installing stuff on. If you have a point in time that you want to save. Simply shut down the vm. make a copy of it. Rename the copy to what you want.
base-arramagong-2-0
This is the base arramagong 2.0.3 application released at FOSS4G 2009. Do not run this version.
arramagong-2-0
This is the arramagong 2.0.3 application released at FOSS4G 2009. You can run this version.


Create Click2Try base Xubuntu image

Tom Callighan from Click2Try created a base Xubuntu system for us. Ask him for a new Xubuntu base image, or ask him to fill in the steps so that we can do it.

Click2Try: Add user=user

The base Xubuntu image only has a root user. The following needs to be done to add user="user", pass="user"

  • In order to allow a low security password, Update /etc/common-password/common-password

Once we place above file then we can create user "user" and set password.

Here are the steps which we have followed. 


#useradd user

#passwd user [Set the password as "user"]
  1. usermod -u 499 user
  1. su user

Above command will create directory once we login using user "user".

About ubuntu release version, yes for gisvm which we build earlier stage was ubuntu904 as per requirement.

Please mail us if you find issues.

Click2Try: Create an Arramagong base image

  • Copy the base-xubuntu-9-10 image, call it arramagong-<version>
  • Make the image external: Highlight the VM in the application manager, shut the VM down (make sure you save the session), click on the sharing tab,

click on the public access button. This feature actually has a second benefit. It keeps the VM running even when you leave the VM from the viewer. That way, when you log back into click2try you can access the VM much quicker.

Click2Try: Create default user

  • Launch the new image you created
  • Log in with: user=root, password=.Same as above.

Creating the GIS Virtual Machine

Set the Version Number and Changes

Update https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/VERSION.txt with the current version number.

Update https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/CHANGES.txt with changes since the last release.

Copy base virtual machine

Copy the base virtual machine to a new directory and rename it to the current version: 

host$ cd /var/lib/vmware/Virtual\ Machines/
host$ cp -pr xubuntu-9.10/ arramagong-3.0-rc2
  • Log into the control panel: https://127.0.0.1:8333/ui
  • Select the base xubuntu image VM: Inventory -> arramagong
  • Select Summary tab
  • In the "Commands" pane, select "Configure VM"
  • Change the hostname from "arramagong" to a name which includes version number. Eg: "arramagong-3.0-rc2"

Alternative name change:

  • You can change the image name in /var/lib/vmware/Virtual\ Machines/arramagong-3.0-rc2/arramagong.vmx . Update the following line as:
displayName = "arramagong-3.0rc2"

Bootstrap the Live DVD

  • Start the Xubuntu Virtual Machine
  • Open a terminal
  • Copy the bootstrap script into your home directory and execute it
arramagong$ cd /tmp
arramagong$ wget https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/bootstrap.sh
arramagong$ chmod a+x bootstrap.sh
arramagong$ sudo ./bootstrap.sh
  • This will install subversion, and the install scripts, and create a link to them from your home directory.

Install all applications

  • Optional: If you have run through this process a number of times, you might be smart enough to have saved a local copy of the tmp/ directory. You can save time and bandwidth by copying the tmp/ contents onto the live DVD:
sudo rsync -avz username@hostname.org:/path_to_tmp_dir/ /tmp/
sudo mkdir /var/cache/apt
sudo cp /tmp/apt/* /var/cache/apt/
  • cd to the install scripts directory, and run the main.sh installer. Note, that we log the output into tee, so that we can search for errors later.
cd ~/gisvm/bin
sudo ./main.sh 2>&1 | tee /var/log/arramagong/main_install.log
  • Answer prompts as they come up, there are a few at the start. Wait for a few hours while everything is downloaded and installed.
  • When finished, and before you shutdown your virtual machine, copy the download directories in /tmp to your local computer. (This is to save you re-downloading the files if you are to run through the process again.)
  • The Live DVD should now be ready to test.

Virtual Machine specific steps

Take a copy of the Virtual Machine, as you don't want VMWare tools installed on the ISO image (in the next step).

Install VMWare tools

VMWare tools allow cut and paste from the host operating system to a VMWare virtual machine, and also speeds up graphic rendering on VMWare.

In the VMWare browser control panel, select the virtual machine, then "Summary" tab. Turn on the virtual machine, then select "Install VMWare tools". This mounts the vmware-tools scripts in the arramagong virtual machine, so that it can be installed. Which you can do using the following commands: 

arramagong$ cd /tmp
arramagong$ tar -zxf /media/cdrom0/VMwareTools-7.7.5-156745.tar.gz
arramagong$ cd vmware-tools-distrib/
arramagong$ sudo ./vmware-install.pl
arramagong$ # answer defaults to all questions

Power off

Then poweroff the virtual machine. (Note that this removes all files in the tmp/ directory). 

arramagong$ sudo halt

Package the VMWare VM

Note: Most of this step is covered in the package.sh script. Check and update the variables in the script before running on an Ubuntu based system.

On the host computer, all temporally files can be removed from the virtual machine folder: log files, ram files, etc. Remove all files EXCEPT: *.vmx (VM definition file) and *.vmdk (virtual disk file)

Shrink the virtual machine: (requires VMWare Server installed) Note: If you get an error from this command, you may need to temporarily rename the *.vmx file before operation and then name it back after 

host$ vmware-vdiskmanager -k *.vmdk

Zip the image up: 

 host$ 7z a -mx=9 ArramagongGISVM2009alpha4.7z ArramagongGISVM2009alpha4/

If the image is greater than 2 Gig, then you also need to split the image. The OSGeo download server isn't configured to accept files of a greater size. 

host$ split -b 1500M ArramagongGISVM2009alpha4.7z ArramagongGISVM2009alpha4.7z

Create the md5sum checksums, so which can be used to confirm that the images have been downloaded correctly: 

host$ md5sum *.7z*

Build the Live DVD ISO image

Build ISO

The ISO will be built as part of the main.sh script and stored in /tmp/remastersys/ . Make sure to copy the ISO onto the osgeo server, and then remove /tmp/remastersys/ or reboot the VM to remove it(must boot up again) before packaging the VM.

  • Run the script to build the ISO, this includes downloading close to 1GB of Windows and Mac installers.
sudo ./build_iso.sh 2>&1 | tee /var/log/arramagong/build_iso.log
  • Once the ISO is complete copy it out to a server (a local server is fastest)
scp /tmp/remastersys/arramagong-livedvd-2.0-final.iso.md5 user@server.org:destination/path/
scp /tmp/remastersys/arramagong-livedvd-2.0-final.iso user@server.org:destination/path/
  • And/OR Wget or scp the file to the upload.osgeo.org server (Note wget is much faster if you have a good webserver to host from)

Backup files

Optional: To save time and bandwidth, it is a good idea to back up all the files downloaded into your /tmp directory and the apt-get cache (var/cache/apt), which you can then copy onto future images you create before starting the build process. These files are copied into /tmp/<version>/ during the build process: 

host$ scp -pr user@<arramagong ip>/tmp/3.1alpha1 arragmagong_tmp_3.1alpha1


Upload to the OSGeo Server

host$ scp -pr arramagong-gisvm-2.0-alpha5 username@upload.osgeo.org:/osgeo/download/livedvd/

Update the index.html file at: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/download/index.html

Check the result at: http://download.osgeo.org/livedvd

Creating a torrent file

Several tools exist for creating torrents. Below are the key settings you need.

Deluge with the torrent creator plugin has been tested.

  • Make sure you have a copy of the file you want to create a torrent for. Mounted remote drives may work.
  • Web Seed - URL to a web server that hosts the file, adds speed to the torrents especially at the start. (Most clients implement this now)
    • Note: Currently OSGeo servers can not handle downloads of files over 2GB, do not web seed from OSGeo for larger files.
  • Tracker - There are 2 open and free trackers that have been tested, use one or the other as most clients do not handle multiple trackers yet.
http://tracker.openbittorrent.com/announce
udp://tracker.openbittorrent.com:80/announce

OR 

http://tracker.publicbt.com:80/announce
udp://tracker.publicbt.com:80/announce
  • Upload the .torrent file of your iso file for others to grab
  • Start the torrent on your machine or dedicated seeder, so that others have somewhere to start.

SVN branches and tags

Naming

  • main development happens in "trunk" (aka HEAD)
  • a branch is split off before release time. Bug fixes (only) happen in it and updates like version 2.1 are tagged from it.
its name might be like "arramagong_2"
  • a tag is a snapshot of the svn at some point in time, aka a release name. It does not make sense to checkin fixes to one. They should ideally be set as read-only after creation to prevent this.
its name might be like "release_20090927_arramagong_2_0"

Creating

cd livedvd/gisvm/

svn copy trunk branches/arramagong_2
svn commit branches/arramagong_2 -m "Splitting off branch for 2.x"

cd branches/arramagong_2/bin/
# something like this, maybe need a for loop or xargs:
sed -e 's+trunk+branches/arramagong_2+' *.sh | less   # check
sed -i -e 's+trunk+branches/arramagong_2+' *.sh       # execute
cd -
 
svn copy branches/arramagong_2 tags/release_20090927_arramagong_2_0
svn commit tags/release_20090927_arramagong_2_0 -m "tag release 2.0"

Merging

To merge a change from trunk into a release branch use "svn merge", as follows: (in this example r2131 from trunk) 

svn up branches/arramagong_2/
cd branches/arramagong_2/

svn merge -c 2131 https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk
svn diff
svn commit -m "bugfix: bikeshed should be orange! (merge from trunk r2131)"

If you do this a lot you might make a little shell script. Here's one called svn_merge_livefromtrunk.sh: 

#!/bin/sh
# Usage: svn_merge_livefromtrunk <rev number>
svn merge -c $1 https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk

See also

Retrieved from "http://wiki.osgeo.org/w/index.php?title=Live_GIS_Build&oldid=46012"