Difference between revisions of "Live GIS Build"

From OSGeo
Jump to navigation Jump to search
m (Modified Lubuntu OS version (18.04 => 22.04) and architecture (i386 => amd64))
 
(221 intermediate revisions by 7 users not shown)
Line 1: Line 1:
−
=About=
+
= Getting started =
  
−
[http://gisvm.com/ GISVM] is an Xubuntu based Virtual Machine which has been installed with a suite of the best Open Source Geospatial software. It is also used to build [http://www.arramagong.com/Arramagong.html Arramagong], the Linux based Live DVD.
+
The first thing you will have to do is check-out the latest build scripts using either one of these two Git servers:
 +
git clone <nowiki>https://git.osgeo.org/gitea/osgeolive/OSGeoLive.git</nowiki>
  
−
=How to add your project or data to GISVM / Arramagong=
+
git clone <nowiki>https://github.com/OSGeo/OSGeoLive.git</nowiki>
  
−
All that's required to add your favorite package into GISVM is to:
+
* see [[Live_GIS_Disc#Git_migration|here]] for more details.
  
−
* '''Recommended''': Subscribe to the [http://lists.osgeo.org/mailman/listinfo/live-demo Live Demo] email list to discuss issues.
+
=How to add your project to OSGeo Live=
  
−
* '''Optional''': Install the latest Arramamgong / GIS virtual machine as per: [[Live GIS Disc Quick Start]].
+
* [[Live_GIS_Add_Project|Instructions for adding new projects]]
−
* write a shell script which installs and configures your stable package into the current GISVM virtual machine (which will usually be the same as installing on Xubuntu or Ubuntu).
 
  
−
* Add the script to subversion, in https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/ , as per http://wiki.osgeo.org/wiki/Live_GIS_Disc#Subversion
+
= Creating a fresh Virtual Machine to use as a build host (Optional) =
−
: If you are unfamiliar with subversion then you can send the script to the [[Live_GIS_Disc#Communication|OSGeo live-demo mailing list]] ''as an attachment'' and we'll see that it makes it in.
+
Refer to: [[Live_GIS_Virtual_Machine]].
  
−
* Notify the GISVM team to test your script, and will ensure to reference your script from ''main.sh''.
+
= Build the Live DVD ISO image =
  
−
== Example scripts ==
 
  
−
You can see example scripts at: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/
+
== Build ISO ==
  
−
The script may be as simple as:
+
This section describes the new method for building OSGeoLive as described in [https://help.ubuntu.com/community/LiveCDCustomization official ubuntu wiki]. This section is self-contained and there is no need to perform any of the procedures described above.
−
[https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/install_mapserver.sh install_mapserver.sh], which just "apt-get install mapserver"
 
  
−
Projects that haven't been packaged yet are slightly more complicated:
+
All you need is a running Ubuntu/Xubuntu/Kubuntu/Lubuntu installation (even within a virtual machine as long as it has ~20GB free disk space). All needed to be done are the following steps under a "user" account:
  
−
The Udig package is a good example: [https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/install_udig.sh install_udig.sh]
+
* Bootstrap the host operating system. If you use the system to build more than once, then this must be done only for the first build
 +
host$ cd /tmp
 +
host$ wget https://github.com/OSGeo/OSGeoLive/raw/master/bin/bootstrap.sh
 +
host$ chmod a+x bootstrap.sh
 +
host$ sudo ./bootstrap.sh
 +
This will install Git and the install scripts, and create a link to them from your home directory.
  
−
* 3rd Party Repositories: Create an <app>.list file with your repository in sources.list.d/ in the svn, in your script copy the file to the local machine like
+
* Set the Version Number and Changes
−
wget https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/sources.list.d/ubuntugis.list \
+
  Update https://github.com/OSGeo/OSGeoLive/blob/master/VERSION.txt with the current version number.
−
  --output-document=/etc/apt/sources.list.d/ubuntugis.list
 
  
−
don't forget to add the repository key like so
+
  Update https://github.com/OSGeo/OSGeoLive/blob/master/CHANGES.txt with changes since the last release.
−
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 68436DDF
+
This list can be a summary of the [https://github.com/OSGeo/OSGeoLive/commits/master revision log] between releases
 +
Commit the changes to Git through a Pull Request right before a release build.
  
−
== Packaging conventions ==
+
* Execute the build script:
 +
host$ cd ~/gisvm/bin
 +
host$ sudo ./build_chroot.sh amd64 release master OSGeo 2>&1 | tee /var/log/osgeolive/chroot-build.log
  
−
* Assume the script is to be executed as root.
+
* Compress the logs:
−
: This means that ~/ is /root, so you should use ~user/ for /home/user.
+
host$ cd ~/livecdtmp
 +
host$ tar czf version-log.tar.gz -C /var/log osgeolive
  
−
* It is ok initially to have manual steps in your script. Make sure you tell the packaging team about it. When this happens, include:
+
* After the completion of the above script the new iso file is located in ~/livecdtmp along with the build logs. In case you wish to rerun the build process, do not remove or move the lubuntu official iso located in this folder to skip downloading it again.
−
echo "install_<package>.sh FIXME: <describe manual step>
 
  
−
* Assume that the script may be executed twice, (as people re-run the install process to get it right).
+
* It is required to reboot your host machine after build is completed
−
** Make sure that nothing is corrupted if the script runs a second time.
 
  
−
* Try not to have an steps which prompt the user for an answer, as we want the scripts to be run automatically. In particular, try running the script a second time, and see if you get prompted to overwrite files.
+
* Once the ISO is complete copy it out to a server (a local server is fastest)
 +
scp ~/livecdtmp/osgeolive-mini-8.0.iso user@server.org:destination/path/
 +
scp ~/livecdtmp/osgeolive-mini-8.0-log.tar.gz user@server.org:destination/path/
  
−
* Copy all downloads into /tmp/build_<package>
+
* And/OR wget or scp the file to the upload.osgeo.org server (Note wget is much faster if you have a good webserver to host from)
−
** Don't delete the /tmp/build_<package> directory. If the script is rerun, then time and bandwidth is saved by not having to download again.
 
  
−
* When downloading large files using wget, use the "-c" continue command, as it will not re-download files if they already exist, and will continue if one is partially downloaded:
+
=== Build Full ISO from Mini ISO ===
−
wget -c package.tar.gz
 
  
−
* Where appropriate, create an icon on the desktop which loads the application.  
+
Any mini iso file resulting from the procedure below can be turned into full iso, which is identical but adds the Windows and Mac software installers. Note that the resulting file may be bigger than a DVD. In that case the list of included applications in the load_mac_installers.sh need to be edited to exclude additional applications.
−
: see <tt>install_qgis.sh</tt> for an example
 
  
−
* Create an icon in the the OSGeo Applications pull down list.
+
The standard current rules used to pick applications:
−
: see <tt>install_qgis.sh</tt> for an example
+
* Desktop Applications
 +
* OSGeo projects and Incubated projects
  
−
* If the application is launched from the browser, then include the application in Firefox bookmarks.
+
sudo ./build_full_iso.sh /full/path/to/osgeo-live-mini-8.0.iso 2>&1 | tee ~/build_full_iso.log
−
:  (Todo: How do we do this?)
 
  
−
== Writing docs ==
+
== How to do development / debugging with the current build method ==
−
Each project should have a distinct '''sentence or two defining''' the project, as a html snippet inside a <nowiki><li>tag</li></nowiki>. There is a template here: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/doc/template_definition.html
 
  
−
Each project should also have up to '''one page description''' about how to use the project on the live DVD. Again, this should be a HTML snippet. See example here: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/doc/template_description.html
+
We have created a debug build process so that projects can now easily create their own iso, including parts of the OSGeoLive (eg only one project) in order to test if the installer scripts work well under this new build method.
 +
Here are the steps to debug/test your application:
  
−
See https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/doc/index.html for prior text examples.
+
===One time steps===
  
−
Once complete, save your docs into <project>_definition.html and <project>_description.html, and email the Live Demo list to ensure that it is included into the install_main_docs.sh script.
+
You will need to create a pure Lubuntu Virtual Machine setup:
  
−
Please use HTML with a limited set of tags. Allowed are the following tags within your paragraph:
+
* Download lubuntu-22.04-desktop-amd64.iso from [http://cdimage.ubuntu.com/lubuntu/releases/22.04/release/lubuntu-22.04-desktop-amd64.iso lubuntu web site].
−
* em (for italic)
+
* Download and install VirtualBox.
−
* strong (for bold)
+
* Create a fresh VM installation of Lubuntu. You will need to create a virtual disk drive with at least 25 GB of space and allocate 768MB of RAM to the VM. During installation set the username to "user" and hostname to "osgeolive". DO NOT install system updates during lubuntu installation or after the installation is done. At this moment we work with the default kernel included in lubuntu.
−
* a (make sure your links work (even in the future))
+
* After the VM is done, login as "user" and open a terminal.
−
* ul
+
* Bootstrap the VM:
−
* li
+
osgeolive$ cd /tmp
−
* h1 for headings
+
osgeolive$ wget https://github.com/OSGeo/OSGeoLive/raw/master/bin/bootstrap.sh
−
* h2
+
osgeolive$ chmod a+x bootstrap.sh
−
* h3
+
osgeolive$ sudo ./bootstrap.sh
  
−
At the moment, we don't plan to include images/logos. That may change in future releases.
+
This will install Git, the install scripts, and create a link to them from your home directory.
  
−
= Build Drivers =
+
===Steps to create the build (repeat as much as needed)===
−
== Which version? ==
 
−
The criteria used to select applications for GISVM is as follows:
 
  
−
# Priority goes to Ubuntu packaged software first, then Debian packaged software (but often those are the same thing). This facilitates easy and reliable maintenance and update. Users will benefit from it. Programmers are encouraged to move their software into [http://wiki.debian.org/DebianGis DebianGIS] or [https://wiki.ubuntu.com/UbuntuGIS UbuntuGIS] repositories.
+
* Make changes to your project's installation script and commit to Git.
−
# Stable, always! GISVM users are mainly starters. So they obviously are better with stable software. They have enough problems already and will gladly be happier without bleeding-edge software bugs.
 
  
−
For people who want the latest version, they can create a modified upgrade script.
+
* Update the git code:
 +
osgeolive$ cd ~/gisvm
 +
osgeolive$ git pull origin master
  
−
== Low Memory ==
+
* Open file inchroot.sh with an editor and comment out all scripts you do not need for your test.  
−
Both Virtual Machines, and a LiveDVD images are likely to be constrained by limited memory. So to reduce memory usage. Disk image size is not of major concern, as we can just distribute less data.
+
osgeolive$ cd ~/gisvm/bin
 +
osgeolive$ vi inchroot.sh
  
−
The following principles should be followed.  
+
* Always leave un-commented the following scripts: setup.sh, install_services.sh, install_mysql.sh, install_java.sh, install_apache2.sh, install_tomcat6.sh, install_desktop.sh and setdown.sh.  
−
# Do not start applications upon power up. (Ie, don't start deamons, allow users to start them instead).
 
−
# Set up examples which, by default, don't depend on other applications. Less applications open, means less memory. Ie, Have GeoServer access a shapefile instead of PostGIS.
 
−
# Try to avoid scenarios which write data to disk, as disk space in the Live DVD is stored in RAM, and is not cleared afterward.
 
  
−
= Directory Structure =
+
* Save your changes and execute the build:
 +
osgeolive$ cd ~/gisvm/bin
 +
osgeolive$ sudo ./build_chroot.sh amd64 2>&1 | tee /var/log/osgeolive/chroot-build.log
  
−
# '''/tmp''' Temporary files (e.g., downloaded archives) go into /tmp. Please create a separate folder for your project.
+
* After a while the iso will be created in ~livecdtmp/
−
# '''/usr/lib''' application are usually installed into /usr/lib
 
−
# '''/usr/bin''' things that get executed by the user such as startup scripts or links to them should go into /usr/bin
 
−
# '''/etc/init.d''' startup/shutdown scripts for services (e.g., postgres, apache, tomcat) are stored in /etc/init.d
 
−
# '''/usr/local/share''' sample data and documentation goes into /usr/local/share
 
−
# '''/etc''' config files are stored in /etc
 
−
# '''/home/user''' user specific config files or working directories can go into <tt>/home/user/</tt>. However, keep files in /home/user as small as possible as this folder is loaded into memory in the Live DVD. Symlinks into /usr/local can be useful here.
 
  
 +
* Do not delete the file ~/livecdtmp/lubuntu-22.04-desktop-amd64.iso as it will be needed for next build (saves time not to download again)
  
−
* ''You are strongly encouraged to use '''/usr/local/''' and '''/var/local/''' for storing non-packaged content.''
+
* Logs are created at /var/log/osgeolive/chroot-build.log
  
−
= Mounting as Existing Image =
+
* Copy the iso and test
−
For the purposes of faster dissemination of updates to the Live image via something like rsync you can mount an image file.
 
−
Make sure you turn off the virtual machine before you attempt to mount it, and that you unmount it before you attempt to run the vm again.
 
  
−
== Using VMWare ==
+
= Build the Live DVD VM image =
−
This method assumes you have vmware server installed, the key is that the vmware-mount script is on your system somewhere.
 
−
This instructions use /space/virtual as the vmware installation folder and /space/virtual/machines as the location of the disk images.
 
−
For more information see the [http://www.vmware.com/pdf/VMwareDiskMount.pdf VMware mounting guide]
 
−
*If it complains about not finding libdir/lib/libcrypto.so.0.9.8/libcrypto.so.0.9.8
 
−
*It's because it's looking for it in your vmware bin/libdir which doesn't exist so symlink
 
−
sudo ln -s /space/virtual/lib/vmware/ libdir
 
−
*Temporarily move your .vmx file out of the directory
 
−
sudo ./vmware-mount -p /space/virtual/machines/gisvm20090828x.vmdk
 
−
*Should be partition 1
 
−
*Make a directory to mount to somewhere easy to find, in my case where I keep my vm images
 
−
mkdir /space/virtual/machines
 
−
*Mount it
 
−
sudo ./vmware-mount /space/virtual/machines/gisvm20090828x.vmdk 1 /space/virtual/machines/mountedimage
 
−
*Don't forget to umount when you're done
 
−
sudo ./vmware-mount -d /space/virtual/machines/mountedimage
 
  
−
== Using VirtualBox ==
+
== Create the VM ==
−
These instructions have not been tested yet, they require VirtualBox 2.0+
 
−
http://forums.virtualbox.org/viewtopic.php?f=7&t=17574
 
  
−
== Using Qemu ==
+
The OSGeoLive Virtual Machine creation process is now exactly similar to a plain Lubuntu VM installation.
 +
Use the mini iso file that was created from the previous chapter.
 +
Instructions can be found [[Live_GIS_Virtual_Machine]]
  
−
Run the following command:
+
== Package the VM ==
−
qemu -cdrom livedvd.iso
 
  
−
== Mount the ISO as a readonly filesystem ==
+
From within the VM, fill empty space with zeros in order to be able to shrink the virtual disk files:
  
−
  mkdir test_iso
+
osgeolive$ sudo ~/gisvm/bin/zerofill.sh
−
  mount -t iso9660 -o loop,ro livedvd.iso ./test_iso
 
  
−
If you want to do this without su rights, use the FuseISO package.
+
Shrink the virtual machine:
−
fuseiso -p livedvd.iso ./test_iso.$$
 
  
−
Unmount the FuseISO filesystem with:
+
  host$ VBoxManage modifyhd osgeolive.vdi --compact
−
  fusermount -u ./test_iso.$$
 
  
−
= Creating a fresh Virtual Machine =
+
Convert to vmdk format (more widely compatible):
−
== System Requirements ==
+
−
* RAM: You can only just get away with 1 Gig. 2 Gig is what you will require to avoid slowdowns due to memory swapping.  
+
host$ VBoxManage clonehd osgeolive.vdi osgeolive-14.0-amd64.vmdk --format VMDK
−
* About 10 Gig of spare hard disk.
+
OR with a recent version of QEMU
 +
host$ qemu-img convert -f vdi -o compat6 -O vmdk osgeolive.vdi osgeolive-14.0-amd64.vmdk
  
−
== Install VMWare Server ==
+
Zip the image up:
−
You will need [http://www.vmware.com/products/server/ VMWare server] which provides all the tools for configuring a VM, setting the disk size, compressing the image etc. VirtualBox doesn't seem to have the compression scripts required, and it doesn't seem to have a way to copy your image to VMWare.
 
  
−
If you are installing on linux, the installer will ask a series of questions. Answer the default for everything, except the default administrator. For this, you need to enter in your <user_id> for the local computer you are installing on.
+
host$ 7z a -mx=9 osgeolive-14.0-amd64.vmdk.7z osgeolive-14.0-amd64.vmdk
  
−
== Create an Xubuntu Virtual Machine ==
+
Create the md5sum checksums, so which can be used to confirm that the images have been downloaded correctly:
−
If you already have an Xubuntu (or similar) Virtual Machine, you can skip this step.
+
host$ md5sum *.7z*
  
−
'''Side note:''' vmbuilder is reported to be a good way to create an Ubuntu image. I haven't tried it, and haven't worked out if it can create the smaller Xubuntu image. (Please update this wiki if you try it). On ubuntu:
+
= Upload the Release =
−
apt-get install python-vm-builder
 
  
−
Otherwise, run VMWare Server, and create an image.
+
== Upload to sourceforge ==
 +
As of 6.0 the official releases are hosted on sourceforge. To upload you need a sourceforge account and permissions to the osgeo-live project upload.
  
−
Download the latest [http://www.xubuntu.org/ Xubuntu ISO]. Copy to the Virtual Machine directory:
+
rsync -e ssh osgeolive-14.0-amd64.iso username,osgeo-live@frs.sourceforge.net:/home/pfs/project/o/os/osgeo-live/14.0/
  
−
wget http://mirror.internode.on.net/pub/ubuntu/xubuntu/9.04/release/xubuntu-9.04-desktop-i386.iso
+
== Upload to the OSGeo Server ==
−
mv xubuntu-9.04-desktop-i386.iso "/var/lib/vmware/Virtual Machines/"
 
  
−
Run vmware, which opens up a browser based User Interface.
+
host$ scp -pr osgeolive-gisvm-2.0-alpha5 username@upload.osgeo.org:/osgeo/download/livedvd/
−
vmware
 
  
−
Create a new Virtual Machine with:
+
Update the index.html file at: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/download/index.html
−
* RAM = 512 Gig (we might want to experiment with less)
 
−
* Hard Disk = 20 Gig (you probably only need 8 Gig or so, but it is a pain to increase later, and can't be done from VMWare Player)
 
−
* Mount the XUbuntu ISO image for the disk drive
 
  
−
Run the Virtual Machine, which will boot from the Xubuntu CD. Select to install Xubuntu, using defaults, plus:
+
Check the result at: http://download.osgeo.org/livedvd
−
* Username = user
 
−
* Password = user
 
  
−
Install vmware tools, as per: http://www.ubuntugeek.com/howto-install-vmware-tools-in-ubuntu.html
+
== Creating a torrent file ==
−
sudo apt-get install build-essential linux-headers-`uname -r` psmisc
+
'''Ibiblio has offered torrent hosting, this section needs to include who to contact/how to setup'''
−
tar xvfz VMwareTools-7.8.6-185404.tar.gz
+
Several tools exist for creating torrents. Below are the key settings you need.
−
cd vmware-tools-distrib/
 
−
sudo ./vmware-install.pl
 
−
# default reply to all questions
 
  
−
You should now have a XUbuntu Virtual Machine image in /var/lib/vmware/Virtual Machines/ . It would be a good idea to save a copy of this machine somewhere, so you can use it again.
+
Deluge with the torrent creator plugin has been tested.
 +
* Make sure you have a copy of the file you want to create a torrent for. Mounted remote drives may work.
 +
* Web Seed - URL to a web server that hosts the file, adds speed to the torrents especially at the start. (Most clients implement this now)
 +
* Tracker - There are 2 open and free trackers that have been tested, use one or the other as most clients do not handle multiple trackers yet.
 +
http://tracker.openbittorrent.com/announce
 +
udp://tracker.openbittorrent.com:80/announce
 +
OR
 +
http://tracker.publicbt.com:80/announce
 +
udp://tracker.publicbt.com:80/announce
 +
* Upload the .torrent file of your iso file for others to grab
 +
* Start the torrent on your machine or dedicated seeder, so that others have somewhere to start.
 +
* There is a possibility of seeding from osgeo machines but a Quality of Service (QoS) system would need to be in place to ensure it does not impact other services.
  
−
= Creating the GIS Virtual Machine =
+
= SVN branches and tags =
  
−
== Set the Version Number and Changes ==
+
=== Naming ===
−
Update https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/VERSION.txt with the current version number.
 
  
−
Update https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/CHANGES.txt with changes since the last release.
+
* main development happens in "<tt>trunk</tt>" (aka HEAD)
 +
* a branch is split off before release time. Bug fixes (only) happen in it and updates like version 2.1 are tagged from it.
 +
: its name might be like "<tt>arramagong_2</tt>"
 +
* a tag is a snapshot of the svn at some point in time, aka a release name. It does not make sense to checkin fixes to one. They should ideally be set as read-only after creation to prevent this.
 +
: its name might be like "<tt>release_20090927_arramagong_2_0</tt>"
  
−
== Bootstrap the Live DVD ==
+
=== Creating ===
−
* Start the Xubuntu Virtual Machine
 
−
* Open a terminal
 
−
* Copy the bootstrap script into your home directory and execute it
 
−
cd ~
 
−
wget https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/bootstrap.sh
 
−
sudo ./bootstrap.sh
 
−
* This will install subversion, and the install scripts, and create a link to them from your home directory.
 
  
−
== Install all applications ==
+
cd livedvd/gisvm/
−
* '''Optional''': If you have run through this process a number of times, you might be smart enough to have saved a local copy of the tmp/ directory. You can save time and bandwidth by copying the tmp/ contents onto the live DVD:
+
−
  sudo rsync -avz username@hostname.org:/path_to_tmp_dir/ /tmp/
+
svn copy trunk branches/arramagong_2
−
  sudo mkdir /var/cache/apt
+
  svn commit branches/arramagong_2 -m "Splitting off branch for 2.x"
−
  sudo mv /tmp/apt/* /var/cache/apt/
+
   
−
* cd to the install scripts directory, and run the main.sh installer. Note, that we log the output into tee, so that we can search for errors later.
+
  cd branches/arramagong_2/bin/
−
  cd ~/gisvm/trunk/bin
+
# something like this, maybe need a for loop or xargs:
−
sudo mkdir /var/log/arramagong/
+
  sed -e 's+gisvm/trunk+gisvm/branches/arramagong_2+g' *.sh | less  # check
−
sudo ./main.sh 2>&1 | tee /var/log/arramagong/main_install.log
+
sed -i -e 's+gisvm/trunk+gisvm/branches/arramagong_2+g' *.sh      # execute
−
* Answer prompts as they come up, there are a few at the start. Wait for a few hours while everything is downloaded and installed.
+
cd -
−
* When finished, and before you shutdown your virtual machine, copy the download directories in /tmp to your local computer. (This is to save you re-downloading the files if you are to run through the process again.)
+
 
−
* The Live DVD should now be ready to test.
+
svn copy branches/arramagong_2 tags/release_20090927_arramagong_2_0_3
 +
svn commit tags/release_20090927_arramagong_2_0_3 -m "tag release 2.0.3"
  
−
== Backup files ==
+
=== Merging ===
−
'''Optional:''' To save time and bandwidth, it is a good idea to back up all the files downloaded into your /tmp directory and the apt-get cache, which you can then copy onto future images you create before starting the build process.
 
−
scp -pr \
 
−
  /tmp \
 
−
  /var/cache/apt \
 
−
user@host.org:/dir/arragmagong_tmp_version
 
  
−
== Compress the image ==
+
To merge a change from <tt>trunk</tt> into a release branch use "<tt>svn&nbsp;merge</tt>", as follows: (in this example r2131 from trunk)
  
−
Clean up all the excess files created during the build process, and remove excess space:
+
svn up branches/arramagong_2/
−
  cd ~/gisvm/trunk/bin
+
  cd branches/arramagong_2/
−
  sudo ./setdown.sh 2>&1 | tee /var/log/arramagong/setdown.log
+
   
 +
svn merge -c 2131 <nowiki>https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk</nowiki>
 +
svn diff
 +
svn commit -m "bugfix: bikeshed should be orange! (merge from trunk r2131)"
  
−
Then poweroff the virtual machine:
+
If you do this a lot you might make a little shell script. Here's one called svn_merge_livefromtrunk.sh:
−
  sudo poweroff
+
  #!/bin/sh
 +
# Usage: svn_merge_livefromtrunk <rev number>
 +
svn merge -c $1 <nowiki>https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk</nowiki>
  
−
== Package the Live DVD ==
+
=== See also ===
  
−
After that, '''on the host computer''', all temporally files can be removed from the virtual machine folder: log files, ram files, etc.
+
* Many good hints can be found in the [https://trac.osgeo.org/grass/wiki/HowToSVN GRASS SVN-help wiki page].
  
−
So, remove all files '''EXCEPT''': *'''.vmx''' (VM definition file) and *'''.vmdk''' (virtual disk file)
+
[[Category: Live-demo]]
  
−
In case you are using VMware software, like VMWare Server or Workstation, you can use the '''vmware-vdiskmanager''' tool for virtual disk file '''shrinking'''.
 
  
−
Just find where this command line tool has been installed on your host computer and run it with the -k option, over the vmdk file.
+
= Staging the new release =
  
−
Example if using VMWare Server on Windows (after a few minutes the vmdk file should reduce aprox. 1 GByte):
+
== Website ==
−
  "%programfiles%\vmware\vmware server\vmware-vdiskmanager.exe" -k "D:\ArramagongGISVM2009alpha4\Arramagong 4.vmdk"
+
Log in to live.osgeo.org.
 +
  cd /osgeo/osgeolive
 +
mkdir ''<versionnumber>''
  
−
+
From another machine (ususally adhoc) go to the root of the docs and do
−
Zip the image up:
+
scp -r /osgeo/livedvd/www/docs/* live.osgeo.org:/osgeo/osgeolive/''<versionumber>''/
−
  7z a -mx=9 ArramagongGISVM2009alpha4.7z ArramagongGISVM2009alpha4/
 
  
−
If the image is greater than 2 Gig, then you also need to split the image. The OSGeo download server isn't configured to accept files of a greater size.
+
Once done check that it's all good:
−
  split -b 1500M ArramagongGISVM2009alpha4.7z ArramagongGISVM2009alpha4.7z
+
  <nowiki>http://live.osgeo.org/archive/</nowiki>''versionnumber''
−
 
 
−
Create the md5sum checksums, so which can be used to confirm that the images have been downloaded correctly:
 
−
md5sum *.7z*
 
  
−
== Upload to the OSGeo Server ==
+
Assuming docs are ok, then update the current symlink:
 +
chmod -R g+w ''<versionnumber>''
 +
rm current
 +
ln -s ''<versionnumber>'' current
  
−
scp -pr arramagong-gisvm-2.0-alpha5 username@upload.osgeo.org:/osgeo/download/livedvd/
+
== Download mirror ==
  
−
Update the index.html file at: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/download/index.html
+
Upload files to https://sourceforge.net/projects/osgeo-live/files/
  
−
Check the result at: http://download.osgeo.org/livedvd
+
Add a new folder for the release, go into it, then scp the file to SourceForge.
 +
* See  https://sourceforge.net/p/forge/documentation/Release%20Files%20for%20Download/#scp
  
−
= Build the Live DVD ISO image =
+
Click the box to stage the files for 3 days to give time for the mirrors to populate.
  
−
[[Category: Live-demo]]
+
Once the files are ready (no longer say "Pending...") and the download links and redirects from download.osgeo.org are tested & working, set the default download on the SourceForge site to either the full or mini ISO by clicking the "i" information button next to the file and "select all" for all platform types, then click the Save button.

Latest revision as of 09:02, 26 June 2022

Getting started

The first thing you will have to do is check-out the latest build scripts using either one of these two Git servers: 

git clone https://git.osgeo.org/gitea/osgeolive/OSGeoLive.git

git clone https://github.com/OSGeo/OSGeoLive.git
  • see here for more details.

How to add your project to OSGeo Live

Creating a fresh Virtual Machine to use as a build host (Optional)

Refer to: Live_GIS_Virtual_Machine.

Build the Live DVD ISO image

Build ISO

This section describes the new method for building OSGeoLive as described in official ubuntu wiki. This section is self-contained and there is no need to perform any of the procedures described above.

All you need is a running Ubuntu/Xubuntu/Kubuntu/Lubuntu installation (even within a virtual machine as long as it has ~20GB free disk space). All needed to be done are the following steps under a "user" account:

  • Bootstrap the host operating system. If you use the system to build more than once, then this must be done only for the first build
host$ cd /tmp
host$ wget https://github.com/OSGeo/OSGeoLive/raw/master/bin/bootstrap.sh
host$ chmod a+x bootstrap.sh
host$ sudo ./bootstrap.sh

This will install Git and the install scripts, and create a link to them from your home directory.

  • Set the Version Number and Changes
 Update https://github.com/OSGeo/OSGeoLive/blob/master/VERSION.txt with the current version number.

 Update https://github.com/OSGeo/OSGeoLive/blob/master/CHANGES.txt with changes since the last release.

This list can be a summary of the revision log between releases Commit the changes to Git through a Pull Request right before a release build.

  • Execute the build script:
host$ cd ~/gisvm/bin
host$ sudo ./build_chroot.sh amd64 release master OSGeo 2>&1 | tee /var/log/osgeolive/chroot-build.log
  • Compress the logs:
host$ cd ~/livecdtmp
host$ tar czf version-log.tar.gz -C /var/log osgeolive
  • After the completion of the above script the new iso file is located in ~/livecdtmp along with the build logs. In case you wish to rerun the build process, do not remove or move the lubuntu official iso located in this folder to skip downloading it again.
  • It is required to reboot your host machine after build is completed
  • Once the ISO is complete copy it out to a server (a local server is fastest)
scp ~/livecdtmp/osgeolive-mini-8.0.iso user@server.org:destination/path/
scp ~/livecdtmp/osgeolive-mini-8.0-log.tar.gz user@server.org:destination/path/
  • And/OR wget or scp the file to the upload.osgeo.org server (Note wget is much faster if you have a good webserver to host from)

Build Full ISO from Mini ISO

Any mini iso file resulting from the procedure below can be turned into full iso, which is identical but adds the Windows and Mac software installers. Note that the resulting file may be bigger than a DVD. In that case the list of included applications in the load_mac_installers.sh need to be edited to exclude additional applications.

The standard current rules used to pick applications:

  • Desktop Applications
  • OSGeo projects and Incubated projects
sudo ./build_full_iso.sh /full/path/to/osgeo-live-mini-8.0.iso 2>&1 | tee ~/build_full_iso.log

How to do development / debugging with the current build method

We have created a debug build process so that projects can now easily create their own iso, including parts of the OSGeoLive (eg only one project) in order to test if the installer scripts work well under this new build method. Here are the steps to debug/test your application:

One time steps

You will need to create a pure Lubuntu Virtual Machine setup:

  • Download lubuntu-22.04-desktop-amd64.iso from lubuntu web site.
  • Download and install VirtualBox.
  • Create a fresh VM installation of Lubuntu. You will need to create a virtual disk drive with at least 25 GB of space and allocate 768MB of RAM to the VM. During installation set the username to "user" and hostname to "osgeolive". DO NOT install system updates during lubuntu installation or after the installation is done. At this moment we work with the default kernel included in lubuntu.
  • After the VM is done, login as "user" and open a terminal.
  • Bootstrap the VM:
osgeolive$ cd /tmp
osgeolive$ wget https://github.com/OSGeo/OSGeoLive/raw/master/bin/bootstrap.sh
osgeolive$ chmod a+x bootstrap.sh
osgeolive$ sudo ./bootstrap.sh

This will install Git, the install scripts, and create a link to them from your home directory.

Steps to create the build (repeat as much as needed)

  • Make changes to your project's installation script and commit to Git.
  • Update the git code:
osgeolive$ cd ~/gisvm
osgeolive$ git pull origin master
  • Open file inchroot.sh with an editor and comment out all scripts you do not need for your test.
osgeolive$ cd ~/gisvm/bin
osgeolive$ vi inchroot.sh
  • Always leave un-commented the following scripts: setup.sh, install_services.sh, install_mysql.sh, install_java.sh, install_apache2.sh, install_tomcat6.sh, install_desktop.sh and setdown.sh.
  • Save your changes and execute the build:
osgeolive$ cd ~/gisvm/bin
osgeolive$ sudo ./build_chroot.sh amd64 2>&1 | tee /var/log/osgeolive/chroot-build.log
  • After a while the iso will be created in ~livecdtmp/
  • Do not delete the file ~/livecdtmp/lubuntu-22.04-desktop-amd64.iso as it will be needed for next build (saves time not to download again)
  • Logs are created at /var/log/osgeolive/chroot-build.log
  • Copy the iso and test

Build the Live DVD VM image

Create the VM

The OSGeoLive Virtual Machine creation process is now exactly similar to a plain Lubuntu VM installation. Use the mini iso file that was created from the previous chapter. Instructions can be found Live_GIS_Virtual_Machine

Package the VM

From within the VM, fill empty space with zeros in order to be able to shrink the virtual disk files: 

osgeolive$ sudo ~/gisvm/bin/zerofill.sh

Shrink the virtual machine: 

host$ VBoxManage modifyhd osgeolive.vdi --compact

Convert to vmdk format (more widely compatible): 

host$ VBoxManage clonehd osgeolive.vdi osgeolive-14.0-amd64.vmdk --format VMDK
OR with a recent version of QEMU
host$ qemu-img convert -f vdi -o compat6 -O vmdk osgeolive.vdi osgeolive-14.0-amd64.vmdk

Zip the image up: 

host$ 7z a -mx=9 osgeolive-14.0-amd64.vmdk.7z osgeolive-14.0-amd64.vmdk

Create the md5sum checksums, so which can be used to confirm that the images have been downloaded correctly: 

host$ md5sum *.7z*

Upload the Release

Upload to sourceforge

As of 6.0 the official releases are hosted on sourceforge. To upload you need a sourceforge account and permissions to the osgeo-live project upload. 

rsync -e ssh osgeolive-14.0-amd64.iso username,osgeo-live@frs.sourceforge.net:/home/pfs/project/o/os/osgeo-live/14.0/

Upload to the OSGeo Server

host$ scp -pr osgeolive-gisvm-2.0-alpha5 username@upload.osgeo.org:/osgeo/download/livedvd/

Update the index.html file at: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/download/index.html

Check the result at: http://download.osgeo.org/livedvd

Creating a torrent file

Ibiblio has offered torrent hosting, this section needs to include who to contact/how to setup Several tools exist for creating torrents. Below are the key settings you need.

Deluge with the torrent creator plugin has been tested.

  • Make sure you have a copy of the file you want to create a torrent for. Mounted remote drives may work.
  • Web Seed - URL to a web server that hosts the file, adds speed to the torrents especially at the start. (Most clients implement this now)
  • Tracker - There are 2 open and free trackers that have been tested, use one or the other as most clients do not handle multiple trackers yet.
http://tracker.openbittorrent.com/announce
udp://tracker.openbittorrent.com:80/announce

OR 

http://tracker.publicbt.com:80/announce
udp://tracker.publicbt.com:80/announce
  • Upload the .torrent file of your iso file for others to grab
  • Start the torrent on your machine or dedicated seeder, so that others have somewhere to start.
  • There is a possibility of seeding from osgeo machines but a Quality of Service (QoS) system would need to be in place to ensure it does not impact other services.

SVN branches and tags

Naming

  • main development happens in "trunk" (aka HEAD)
  • a branch is split off before release time. Bug fixes (only) happen in it and updates like version 2.1 are tagged from it.
its name might be like "arramagong_2"
  • a tag is a snapshot of the svn at some point in time, aka a release name. It does not make sense to checkin fixes to one. They should ideally be set as read-only after creation to prevent this.
its name might be like "release_20090927_arramagong_2_0"

Creating

cd livedvd/gisvm/

svn copy trunk branches/arramagong_2
svn commit branches/arramagong_2 -m "Splitting off branch for 2.x"

cd branches/arramagong_2/bin/
# something like this, maybe need a for loop or xargs:
sed -e 's+gisvm/trunk+gisvm/branches/arramagong_2+g' *.sh | less   # check
sed -i -e 's+gisvm/trunk+gisvm/branches/arramagong_2+g' *.sh       # execute
cd -
 
svn copy branches/arramagong_2 tags/release_20090927_arramagong_2_0_3
svn commit tags/release_20090927_arramagong_2_0_3 -m "tag release 2.0.3"

Merging

To merge a change from trunk into a release branch use "svn merge", as follows: (in this example r2131 from trunk) 

svn up branches/arramagong_2/
cd branches/arramagong_2/

svn merge -c 2131 https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk
svn diff
svn commit -m "bugfix: bikeshed should be orange! (merge from trunk r2131)"

If you do this a lot you might make a little shell script. Here's one called svn_merge_livefromtrunk.sh: 

#!/bin/sh
# Usage: svn_merge_livefromtrunk <rev number>
svn merge -c $1 https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk

See also


Staging the new release

Website

Log in to live.osgeo.org. 

cd /osgeo/osgeolive
mkdir <versionnumber>

From another machine (ususally adhoc) go to the root of the docs and do 

scp -r /osgeo/livedvd/www/docs/* live.osgeo.org:/osgeo/osgeolive/<versionumber>/

Once done check that it's all good: 

http://live.osgeo.org/archive/versionnumber

Assuming docs are ok, then update the current symlink: 

chmod -R g+w <versionnumber>
rm current
ln -s <versionnumber> current

Download mirror

Upload files to https://sourceforge.net/projects/osgeo-live/files/

Add a new folder for the release, go into it, then scp the file to SourceForge.

Click the box to stage the files for 3 days to give time for the mirrors to populate.

Once the files are ready (no longer say "Pending...") and the download links and redirects from download.osgeo.org are tested & working, set the default download on the SourceForge site to either the full or mini ISO by clicking the "i" information button next to the file and "select all" for all platform types, then click the Save button.

Retrieved from "http://wiki.osgeo.org/w/index.php?title=Live_GIS_Build&oldid=128684"