Live GIS Build

From OSGeo
Revision as of 10:28, 16 September 2009 by Wiki-HamishBowman (talk | contribs) (pkg table)
Jump to navigation Jump to search

About

GISVM is an Xubuntu based Virtual Machine which has been installed with a suite of the best Open Source Geospatial software. It is also used to build Arramagong, the Linux based Live DVD.

How to add your project or data to GISVM / Arramagong

All that's required to add your favorite package into GISVM is to:

  • Recommended: Subscribe to the Live Demo email list to discuss issues.
  • Optional: Install the latest Arramamgong / GIS virtual machine as per: Live GIS Disc Quick Start.
  • write a shell script which installs and configures your stable package into the current GISVM virtual machine (which will usually be the same as installing on Xubuntu or Ubuntu).
If you are unfamiliar with subversion then you can send the script to the OSGeo live-demo mailing list as an attachment and we'll see that it makes it in.
  • Notify the GISVM team to test your script, and will ensure to reference your script from main.sh.

Example scripts

You can see example scripts at: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/

The script may be as simple as: install_mapserver.sh, which just "apt-get install mapserver"

Projects that haven't been packaged yet are slightly more complicated:

The Udig package is a good example: install_udig.sh

  • 3rd Party Repositories: Create an <app>.list file with your repository in sources.list.d/ in the svn, in your script copy the file to the local machine like
wget https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/sources.list.d/ubuntugis.list \
  --output-document=/etc/apt/sources.list.d/ubuntugis.list

don't forget to add the repository key like so 

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 68436DDF

Packaging conventions

  • Assume the script is to be executed as root.
This means that ~/ is /root, so you should use ~user/ for /home/user.
  • It is ok initially to have manual steps in your script. Make sure you tell the packaging team about it. When this happens, include:
echo "install_<package>.sh FIXME: <describe manual step>
  • Assume that the script may be executed twice, (as people re-run the install process to get it right).
    • Make sure that nothing is corrupted if the script runs a second time.
  • Try not to have an steps which prompt the user for an answer, as we want the scripts to be run automatically. In particular, try running the script a second time, and see if you get prompted to overwrite files.
  • Copy all downloads into /tmp/build_<package>
    • Don't delete the /tmp/build_<package> directory. If the script is rerun, then time and bandwidth is saved by not having to download again.
  • When downloading large files using wget, use the "-c" continue command, as it will not re-download files if they already exist, and will continue if one is partially downloaded:
wget -c package.tar.gz
  • Where appropriate, create an icon on the desktop which loads the application.
see install_qgis.sh for an example
  • Create an icon in the the OSGeo Applications pull down list.
see install_qgis.sh for an example
  • If the application is launched from the browser, then include the application in Firefox bookmarks.
(Todo: How do we do this?)

Writing docs

Each project should have a distinct sentence or two defining the project, as a html snippet inside a <li>tag</li>. There is a template here: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/doc/template_definition.html

Each project should also have up to one page description about how to use the project on the live DVD. Again, this should be a HTML snippet. See example here: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/doc/template_description.html

See https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/doc/index.html for prior text examples.

Once complete, save your docs into <project>_definition.html and <project>_description.html, and email the Live Demo list to ensure that it is included into the install_main_docs.sh script.

Please use HTML with a limited set of tags. Allowed are the following tags within your paragraph:

  • em (for italic)
  • strong (for bold)
  • a (make sure your links work (even in the future))
  • ul
  • li
  • h1 for headings
  • h2
  • h3

At the moment, we don't plan to include images/logos. That may change in future releases.


Adding to the Package list

Once your application is added in SVN please make sure the information about it is up to date on the Live GIS Disc Packages table.

Build Drivers

Which version?

The criteria used to select applications for GISVM is as follows:

  1. Priority goes to Ubuntu packaged software first, then Debian packaged software (but often those are the same thing). This facilitates easy and reliable maintenance and update. Users will benefit from it. Programmers are encouraged to move their software into DebianGIS or UbuntuGIS repositories.
  2. Stable, always! GISVM users are mainly starters. So they obviously are better with stable software. They have enough problems already and will gladly be happier without bleeding-edge software bugs.

For people who want the latest version, they can create a modified upgrade script.

Low Memory

Both Virtual Machines, and a LiveDVD images are likely to be constrained by limited memory. So to reduce memory usage. Disk image size is not of major concern, as we can just distribute less data.

The following principles should be followed.

  1. Do not start applications upon power up. (Ie, don't start deamons, allow users to start them instead).
  2. Set up examples which, by default, don't depend on other applications. Less applications open, means less memory. Ie, Have GeoServer access a shapefile instead of PostGIS.
  3. Try to avoid scenarios which write data to disk, as disk space in the Live DVD is stored in RAM, and is not cleared afterward.

Directory Structure

  1. /tmp Temporary files (e.g., downloaded archives) go into /tmp. Please create a separate folder for your project.
  2. /usr/lib application are usually installed into /usr/lib
  3. /usr/bin things that get executed by the user such as startup scripts or links to them should go into /usr/bin
  4. /etc/init.d startup/shutdown scripts for services (e.g., postgres, apache, tomcat) are stored in /etc/init.d
  5. /usr/local/share sample data and documentation goes into /usr/local/share
  6. /etc config files are stored in /etc
  7. /home/user user specific config files or working directories can go into /home/user/. However, keep files in /home/user as small as possible as this folder is loaded into memory in the Live DVD. Symlinks into /usr/local can be useful here.


  • You are strongly encouraged to use /usr/local/ and /var/local/ for storing non-packaged content.

Mounting as Existing Image

For the purposes of faster dissemination of updates to the Live image via something like rsync you can mount an image file. Make sure you turn off the virtual machine before you attempt to mount it, and that you unmount it before you attempt to run the vm again.

Using VMWare

This method assumes you have vmware server installed, the key is that the vmware-mount script is on your system somewhere. This instructions use /space/virtual as the vmware installation folder and /space/virtual/machines as the location of the disk images. For more information see the VMware mounting guide

  • If it complains about not finding libdir/lib/libcrypto.so.0.9.8/libcrypto.so.0.9.8
  • It's because it's looking for it in your vmware bin/libdir which doesn't exist so symlink
sudo ln -s /space/virtual/lib/vmware/ libdir
  • Temporarily move your .vmx file out of the directory
sudo ./vmware-mount -p /space/virtual/machines/gisvm20090828x.vmdk
  • Should be partition 1
  • Make a directory to mount to somewhere easy to find, in my case where I keep my vm images
mkdir /space/virtual/machines
  • Mount it
sudo ./vmware-mount /space/virtual/machines/gisvm20090828x.vmdk 1 /space/virtual/machines/mountedimage
  • Don't forget to umount when you're done
sudo ./vmware-mount -d /space/virtual/machines/mountedimage

Using VirtualBox

These instructions have not been tested yet, they require VirtualBox 2.0+ http://forums.virtualbox.org/viewtopic.php?f=7&t=17574

Using Qemu

Run the following command: 

qemu -cdrom livedvd.iso

Mount the ISO as a readonly filesystem

 mkdir test_iso
 mount -t iso9660 -o loop,ro livedvd.iso ./test_iso

If you want to do this without su rights, use the FuseISO package. 

fuseiso -p livedvd.iso ./test_iso.$$

Unmount the FuseISO filesystem with: 

fusermount -u ./test_iso.$$

Creating a fresh Virtual Machine

System Requirements

  • RAM: You can only just get away with 1 Gig. 2 Gig is what you will require to avoid slowdowns due to memory swapping.
  • About 10 Gig of spare hard disk.

Install VMWare Server

You will need VMWare server which provides all the tools for configuring a VM, setting the disk size, compressing the image etc. VirtualBox doesn't seem to have the compression scripts required, and it doesn't seem to have a way to copy your image to VMWare.

If you are installing on linux, the installer will ask a series of questions. Answer the default for everything, except the default administrator. For this, you need to enter in your <user_id> for the local computer you are installing on.

Create an Xubuntu Virtual Machine

If you already have an Xubuntu (or similar) Virtual Machine, you can skip this step.

Side note: vmbuilder is reported to be a good way to create an Ubuntu image. I haven't tried it, and haven't worked out if it can create the smaller Xubuntu image. (Please update this wiki if you try it). On ubuntu: 

apt-get install python-vm-builder

Otherwise, run VMWare Server, and create an image.

Download the latest Xubuntu ISO. Copy to the Virtual Machine directory: 

wget http://mirror.internode.on.net/pub/ubuntu/xubuntu/9.04/release/xubuntu-9.04-desktop-i386.iso
mv xubuntu-9.04-desktop-i386.iso "/var/lib/vmware/Virtual Machines/"

Run vmware, which opens up a browser based User Interface. 

vmware

Create a new Virtual Machine with:

  • RAM = 512 Gig (we might want to experiment with less)
  • Hard Disk = 20 Gig (you probably only need 8 Gig or so, but it is a pain to increase later, and can't be done from VMWare Player)
  • Mount the XUbuntu ISO image for the disk drive

Run the Virtual Machine, which will boot from the Xubuntu CD. Select to install Xubuntu, using defaults, plus:

  • Username = user
  • Password = user

Install vmware tools, as per: http://www.ubuntugeek.com/howto-install-vmware-tools-in-ubuntu.html 

sudo apt-get install build-essential linux-headers-`uname -r` psmisc
tar xvfz VMwareTools-7.8.6-185404.tar.gz
cd vmware-tools-distrib/
sudo ./vmware-install.pl
# default reply to all questions

You should now have a XUbuntu Virtual Machine image in /var/lib/vmware/Virtual Machines/ . It would be a good idea to save a copy of this machine somewhere, so you can use it again.

Creating the GIS Virtual Machine

Set the Version Number and Changes

Update https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/VERSION.txt with the current version number.

Update https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/CHANGES.txt with changes since the last release.

Bootstrap the Live DVD

  • Start the Xubuntu Virtual Machine
  • Open a terminal
  • Copy the bootstrap script into your home directory and execute it
cd ~
wget https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/bootstrap.sh
sudo ./bootstrap.sh
  • This will install subversion, and the install scripts, and create a link to them from your home directory.

Install all applications

  • Optional: If you have run through this process a number of times, you might be smart enough to have saved a local copy of the tmp/ directory. You can save time and bandwidth by copying the tmp/ contents onto the live DVD:
sudo rsync -avz username@hostname.org:/path_to_tmp_dir/ /tmp/
sudo mkdir /var/cache/apt
sudo mv /tmp/apt/* /var/cache/apt/
  • cd to the install scripts directory, and run the main.sh installer. Note, that we log the output into tee, so that we can search for errors later.
cd ~/gisvm/trunk/bin
sudo mkdir /var/log/arramagong/
sudo ./main.sh 2>&1 | tee /var/log/arramagong/main_install.log
  • Answer prompts as they come up, there are a few at the start. Wait for a few hours while everything is downloaded and installed.
  • When finished, and before you shutdown your virtual machine, copy the download directories in /tmp to your local computer. (This is to save you re-downloading the files if you are to run through the process again.)
  • The Live DVD should now be ready to test.

Backup files

Optional: To save time and bandwidth, it is a good idea to back up all the files downloaded into your /tmp directory and the apt-get cache, which you can then copy onto future images you create before starting the build process. 

scp -pr \
  /tmp \
  /var/cache/apt \
user@host.org:/dir/arragmagong_tmp_version

Compress the image

Clean up all the excess files created during the build process, and remove excess space: 

cd ~/gisvm/trunk/bin
sudo ./setdown.sh 2>&1 | tee /var/log/arramagong/setdown.log

Then poweroff the virtual machine: 

sudo poweroff

Package the Live DVD

After that, on the host computer, all temporally files can be removed from the virtual machine folder: log files, ram files, etc.

So, remove all files EXCEPT: *.vmx (VM definition file) and *.vmdk (virtual disk file)

In case you are using VMware software, like VMWare Server or Workstation, you can use the vmware-vdiskmanager tool for virtual disk file shrinking.

Just find where this command line tool has been installed on your host computer and run it with the -k option, over the vmdk file.

Example if using VMWare Server on Windows (after a few minutes the vmdk file should reduce aprox. 1 GByte): 

"%programfiles%\vmware\vmware server\vmware-vdiskmanager.exe" -k "D:\ArramagongGISVM2009alpha4\Arramagong 4.vmdk"


Zip the image up: 

 7z a -mx=9 ArramagongGISVM2009alpha4.7z ArramagongGISVM2009alpha4/

If the image is greater than 2 Gig, then you also need to split the image. The OSGeo download server isn't configured to accept files of a greater size. 

split -b 1500M ArramagongGISVM2009alpha4.7z ArramagongGISVM2009alpha4.7z

Create the md5sum checksums, so which can be used to confirm that the images have been downloaded correctly: 

md5sum *.7z*

Upload to the OSGeo Server

scp -pr arramagong-gisvm-2.0-alpha5 username@upload.osgeo.org:/osgeo/download/livedvd/

Update the index.html file at: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/download/index.html

Check the result at: http://download.osgeo.org/livedvd

Build the Live DVD ISO image

Retrieved from "http://wiki.osgeo.org/w/index.php?title=Live_GIS_Build&oldid=41018"