Difference between revisions of "Live GIS Build"

From OSGeo
Jump to navigation Jump to search
m (Modified Lubuntu OS version (18.04 => 22.04) and architecture (i386 => amd64))
 
(77 intermediate revisions by 7 users not shown)
Line 1: Line 1:
−
=About=
+
= Getting started =
  
−
* See the [[Live GIS Disc]] for the main project page
+
The first thing you will have to do is check-out the latest build scripts using either one of these two Git servers:
 +
git clone <nowiki>https://git.osgeo.org/gitea/osgeolive/OSGeoLive.git</nowiki>
  
−
[http://www.arramagong.com/Arramagong.html Arramagong] is an [http://www.xubuntu.org xUbuntu] based Virtual Machine and Live DVD which has been installed with a suite of the best Open Source Geospatial software.
+
git clone <nowiki>https://github.com/OSGeo/OSGeoLive.git</nowiki>
  
−
=How to add your project or data to the LiveGIS Disc/VM=
+
* see [[Live_GIS_Disc#Git_migration|here]] for more details.
  
−
All that's required to add your favorite package into the build is to:
+
=How to add your project to OSGeo Live=
  
−
* '''Recommended''': Subscribe to the [http://lists.osgeo.org/mailman/listinfo/live-demo Live Demo] email list to discuss issues.
+
* [[Live_GIS_Add_Project|Instructions for adding new projects]]
  
−
* '''Optional''': Install the latest Arramamgong / GIS virtual machine as per: [[Live GIS Disc Quick Start]].
+
= Creating a fresh Virtual Machine to use as a build host (Optional) =
−
* Write a shell script which installs and configures your stable package into the current Live GIS virtual machine (which will usually be the same as installing on Xubuntu or Ubuntu).
+
Refer to: [[Live_GIS_Virtual_Machine]].
−
: ''All install scripts for building the LiveGIS disc shall be licensed as LGPL.''
 
−
* Add the script to subversion, in https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/ , as per http://wiki.osgeo.org/wiki/Live_GIS_Disc#Subversion
 
−
: If you are unfamiliar with subversion then you can send the script to the [[Live_GIS_Disc#Communication|OSGeo live-demo mailing list]] ''as an attachment'' and we'll see that it makes it in.
 
−
* Notify the LiveGIS team to test your script, and will ensure to reference your script from ''main.sh''.
 
  
−
== Getting started ==
+
= Build the Live DVD ISO image =
  
−
The first think you will want to do is check out a copy of the existing build scripts using Subversion:
 
−
svn co <nowiki>https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk</nowiki> osgeo_live/trunk/
 
  
−
== Example scripts ==
+
== Build ISO ==
  
−
You can view example scripts at:
+
This section describes the new method for building OSGeoLive as described in [https://help.ubuntu.com/community/LiveCDCustomization official ubuntu wiki]. This section is self-contained and there is no need to perform any of the procedures described above.
−
: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/
 
  
−
The script may be as simple as:
+
All you need is a running Ubuntu/Xubuntu/Kubuntu/Lubuntu installation (even within a virtual machine as long as it has ~20GB free disk space). All needed to be done are the following steps under a "user" account:
−
[https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/install_mapserver.sh install_mapserver.sh], which just
 
−
apt-get install mapserver
 
  
−
Projects that haven't been packaged for Ubuntu yet are slightly more complicated:
+
* Bootstrap the host operating system. If you use the system to build more than once, then this must be done only for the first build
 +
host$ cd /tmp
 +
host$ wget https://github.com/OSGeo/OSGeoLive/raw/master/bin/bootstrap.sh
 +
host$ chmod a+x bootstrap.sh
 +
host$ sudo ./bootstrap.sh
 +
This will install Git and the install scripts, and create a link to them from your home directory.
  
−
The uDig package is a good example: [https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/install_udig.sh install_udig.sh]
+
* Set the Version Number and Changes
 +
  Update https://github.com/OSGeo/OSGeoLive/blob/master/VERSION.txt with the current version number.
  
−
* 3rd Party Repositories: Create an <app>.list file with your repository in sources.list.d/ in the svn, in your script copy the file to the local machine like
+
  Update https://github.com/OSGeo/OSGeoLive/blob/master/CHANGES.txt with changes since the last release.
−
cp -f ../sources.list/ubuntugis.list /etc/apt/sources.list.d/
+
This list can be a summary of the [https://github.com/OSGeo/OSGeoLive/commits/master revision log] between releases
 +
Commit the changes to Git through a Pull Request right before a release build.
  
−
don't forget to add the repository key like so
+
* Execute the build script:
−
  apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 68436DDF
+
  host$ cd ~/gisvm/bin
 +
host$ sudo ./build_chroot.sh amd64 release master OSGeo 2>&1 | tee /var/log/osgeolive/chroot-build.log
  
−
== Packaging conventions ==
+
* Compress the logs:
 +
host$ cd ~/livecdtmp
 +
host$ tar czf version-log.tar.gz -C /var/log osgeolive
  
−
* '''sudo''': Assume the script is to be executed as root. This means that sudo is only needed if you need to execute a command as e.g. the database user.
+
* After the completion of the above script the new iso file is located in ~/livecdtmp along with the build logs. In case you wish to rerun the build process, do not remove or move the lubuntu official iso located in this folder to skip downloading it again.
−
: This means that ~/ refers to the directory /home/root/, so you should use ~user/ to reference /home/user/.
 
−
: Preferred method is to put the <tt>USER_NAME</tt> and <tt>USER_HOME</tt> variables at the top, so these can be easily changed in all scripts if needed.
 
−
USER_NAME="user"
 
−
USER_HOME="/home/$USER_NAME"
 
−
cp package_readme.txt "$USER_HOME"
 
  
−
* '''Called twice''': Assume that the script will be executed multiple times, (as people may re-run the install process to get it right).
+
* It is required to reboot your host machine after build is completed
−
: Make sure that nothing is corrupted if the script runs a second time.
 
  
−
* '''No Prompts''': Try not to have any interactive steps which prompt the user for an answer, as we want the scripts to be run automatically. In particular, try running the script a second time, and see if you get prompted to overwrite files.
+
* Once the ISO is complete copy it out to a server (a local server is fastest)
−
 
+
scp ~/livecdtmp/osgeolive-mini-8.0.iso user@server.org:destination/path/
−
* '''/tmp''': Copy all downloads into /tmp/build_<package>/
+
scp ~/livecdtmp/osgeolive-mini-8.0-log.tar.gz user@server.org:destination/path/
−
: Don't delete the /tmp/build_<package> directory afterwards. If the script is re-run, then time and bandwidth is saved by not having to download large files again.
 
−
 
 
−
* '''Icons''': Where appropriate, create an icon on the desktop which loads the application. The file should be called <packagename>.desktop and placed in /home/user/Desktop/ for further processing. The icons will be sorted and the Geospatial menu will be auto-generated from here.
 
−
: Install <packagename>.png icon files to the /usr/share/icons/ directory.
 
−
: see <tt>install_qgis.sh</tt> for an example
 
−
 
 
−
* '''Browser bookmarks:''' If the application is launched from the browser, then include the application in Firefox bookmarks.
 
−
:  (Todo: How do we do this?)
 
  
−
* '''wget''': When downloading large files using wget, use the "-c" continue command, as it will not re-download files if they already exist, and will continue if one is partially downloaded:
+
* And/OR wget or scp the file to the upload.osgeo.org server (Note wget is much faster if you have a good webserver to host from)
−
wget -c package.tar.gz
 
−
: Do not use -c if there is a chance the file of that name will change on the server. Add "<tt>--progress=dot:mega</tt>" to the wget command line if the file is at all large.
 
−
: Tiny files should use wget's -nv (non-verbose) flag.
 
  
−
* '''mkdir''': Use -p (create parents too) option when creating a directory, as it doesn't complain about creating the directory a second time.
+
=== Build Full ISO from Mini ISO ===
−
mkdir -p <dir>
 
  
−
* '''Binary files''': Do not put large (>5mb) binary files into the SVN repository without asking first. Due to SVN's preservation of history, once they are added they can never be removed, and forever-more clog up the backend database. Subversion is designed to be a ''source code management'' tool, not a data warehouse. There is space for large static data files (e.g. sample data) at http://download.osgeo.org/livedvd/data/. Just ask on the mailing list if you'd like to house something there.
+
Any mini iso file resulting from the procedure below can be turned into full iso, which is identical but adds the Windows and Mac software installers. Note that the resulting file may be bigger than a DVD. In that case the list of included applications in the load_mac_installers.sh need to be edited to exclude additional applications.  
  
−
== Writing docs ==
+
The standard current rules used to pick applications:
−
=== Definition document ===
+
* Desktop Applications
−
Each project is to have a distinct ''one or two sentences'' defining the project, as a html snippet inside a <nowiki><li>tag</li></nowiki>. <!-- technically </li> isn't really needed -->
+
* OSGeo projects and Incubated projects
−
There is a template here: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/doc/template_definition.html.
 
  
−
The debian package title should provide a suitable description for .deb packaged projects. Provided by:
+
sudo ./build_full_iso.sh /full/path/to/osgeo-live-mini-8.0.iso 2>&1 | tee ~/build_full_iso.log
−
dpkg -s $PACKAGE | grep '^Description:' | cut -f2- -d' '
 
  
−
Save the file here: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/doc/descriptions/<project>_definition.html
+
== How to do development / debugging with the current build method ==
  
−
These files will be later assembled into the master disc contents help page.
+
We have created a debug build process so that projects can now easily create their own iso, including parts of the OSGeoLive (eg only one project) in order to test if the installer scripts work well under this new build method.
 +
Here are the steps to debug/test your application:
  
−
=== Description document ===
+
===One time steps===
−
: (''how best to do this is the subject of ongoing discussion, check on the mailing list'')
 
−
Each project is to also have up to ''one page description'' about how to use the project on the live DVD, saved in the .odt format (which can be edited with Open Office or AbiWord). See example here: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/doc/template_description.odt
 
  
−
Once complete, save your docs into <project>_description.odt, and email the Live-demo list to ensure it is reviewed by the community. This file will be converted to html format as part of the build process.
+
You will need to create a pure Lubuntu Virtual Machine setup:
  
−
You may include links to locally installed or remote pdf or html manuals, but keep in mind that there is a good chance that the user will not have internet access.
+
* Download lubuntu-22.04-desktop-amd64.iso from [http://cdimage.ubuntu.com/lubuntu/releases/22.04/release/lubuntu-22.04-desktop-amd64.iso lubuntu web site].
 +
* Download and install VirtualBox.
 +
* Create a fresh VM installation of Lubuntu. You will need to create a virtual disk drive with at least 25 GB of space and allocate 768MB of RAM to the VM. During installation set the username to "user" and hostname to "osgeolive". DO NOT install system updates during lubuntu installation or after the installation is done. At this moment we work with the default kernel included in lubuntu.
 +
* After the VM is done, login as "user" and open a terminal.
 +
* Bootstrap the VM:
 +
osgeolive$ cd /tmp
 +
osgeolive$ wget https://github.com/OSGeo/OSGeoLive/raw/master/bin/bootstrap.sh
 +
osgeolive$ chmod a+x bootstrap.sh
 +
osgeolive$ sudo ./bootstrap.sh
  
−
Save the file here: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/doc/descriptions/project_definition.odt
+
This will install Git, the install scripts, and create a link to them from your home directory.
  
−
=== Licence document ===
+
===Steps to create the build (repeat as much as needed)===
−
Each project should create one table row for each licence offered by the project. A template for the file to be produced is here: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/doc/template_licence.html
 
  
−
Save the file here: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/doc/descriptions/project_licence.html
+
* Make changes to your project's installation script and commit to Git.
  
−
Please also include the software license info in the package_description file.
+
* Update the git code:
 +
osgeolive$ cd ~/gisvm
 +
osgeolive$ git pull origin master
  
−
== Add to the master project list ==
+
* Open file inchroot.sh with an editor and comment out all scripts you do not need for your test.
 +
osgeolive$ cd ~/gisvm/bin
 +
osgeolive$ vi inchroot.sh
  
−
Once your application is added in SVN please make sure the information about it is up to date on the status table at: http://spreadsheets.google.com/ccc?key=0Al9zh8DjmU_RdGIzd0VLLTBpQVJuNVlHMlBWSDhKLXc&hl=en_GB
+
* Always leave un-commented the following scripts: setup.sh, install_services.sh, install_mysql.sh, install_java.sh, install_apache2.sh, install_tomcat6.sh, install_desktop.sh and setdown.sh.  
  
−
== Testing procedure ==
+
* Save your changes and execute the build:
 +
osgeolive$ cd ~/gisvm/bin
 +
osgeolive$ sudo ./build_chroot.sh amd64 2>&1 | tee /var/log/osgeolive/chroot-build.log
  
−
Provide test steps for your application as per the [[Live GIS Disc Testing]] wiki page.
+
* After a while the iso will be created in ~livecdtmp/
  
−
Once the application has been included into a release candidate, ensure the test steps are executed and work on the release candidate. Write up the results as per the [[Live GIS Disc Testing]] wiki page.
+
* Do not delete the file ~/livecdtmp/lubuntu-22.04-desktop-amd64.iso as it will be needed for next build (saves time not to download again)
  
−
= Packaging considerations =
+
* Logs are created at /var/log/osgeolive/chroot-build.log
  
−
The criteria used to select applications for the LiveGIS follow.
+
* Copy the iso and test
  
−
== Technical limitations ==
+
= Build the Live DVD VM image =
−
=== Space considerations ===
 
−
* DVD: approx 4.2gb compressed
 
−
* USB stick: For a 4gb persistent USB stick with some room left over for user-workspace, 3.2-3.5gb compressed. (This pretty much works out to the DVD version minus the preloaded Mac and Windows installers)
 
−
* Virtual Machines: theoretically unlimited, but in practice 20gb uncompressed
 
  
−
So smaller apps (<30mb) of lesser priority may easily slip in under the ire radar, but an external project that wants to use a 25% of the available disc space would be the first to be compromised away.
+
== Create the VM ==
  
−
=== Memory considerations ===
+
The OSGeoLive Virtual Machine creation process is now exactly similar to a plain Lubuntu VM installation.
−
Both Virtual Machines, and a LiveDVD images are likely to be constrained by limited memory. So to reduce memory usage. Disk image size is not of major concern, as we can just distribute less data.
+
Use the mini iso file that was created from the previous chapter.  
 +
Instructions can be found [[Live_GIS_Virtual_Machine]]
  
−
The following principles should be followed.
+
== Package the VM ==
−
* Do not start applications upon power up. (Ie, don't start deamons, allow users to start them instead).
 
−
* Set up examples which, by default, don't depend on other applications. Less applications open, means less memory. Ie, Have GeoServer access a shapefile instead of PostGIS.
 
−
* Try to avoid scenarios which write data to disk, as disk space in the Live DVD is stored in RAM, and is not cleared afterward.
 
  
−
File permissions and filesystem user IDs can get garbled during the ISO build process. Therefore a special method is used to ensure that data files which must be read-write can be accessed.
+
From within the VM, fill empty space with zeros in order to be able to shrink the virtual disk files:
  
−
To install a writable data directory or files, change its group to "users" and set the file(s) as group-writable:
+
osgeolive$ sudo ~/gisvm/bin/zerofill.sh
  
−
adduser $USER_NAME users
+
Shrink the virtual machine:
−
chmod -R g+w /usr/local/share/<packagename>/data/
 
−
chown -R root.users /usr/local/share/<packagename>/data/
 
  
−
and symlink into the user's home directory if necessary.
+
host$ VBoxManage modifyhd osgeolive.vdi --compact
  
−
Setting files to have universal write permission is strongly discouraged.
+
Convert to vmdk format (more widely compatible):
−
 
 
−
== What gets on the disc? ==
 
−
 
 
−
* OSGeo projects and projects-in-incubation and their sample data + tutorials are given first priority. This will be followed by software needed by the [[FOSS4G]] conference Workshops and Tutorial sessions, and projects already involved with [[OSGeo Labs]].
 
−
 
 
−
External projects may be given a spot on the disc as space allows. If an installer has to be disabled or sample data cut down, this will be done by name in the main build script, the package's install script will remain on the disc in case the user wants to add the extra package by hand, or to allow other packagers to create a remix disc with minimal effort.
 
−
 
 
−
* External projects must be Open Source*, and must have a Geospatial component.
 
−
 
 
−
: [*] ''(i.e. using an OSI approved license and be freely redistributable)''
 
−
 
 
−
Your project should designate an official contact person and script maintainer. We will help as we can, but in the end it is the individual project's responsibility to update and maintain your install script as versions/URLs evolve. If a package remains broken by the time the release-candidate builds are being produced/tested and the script maintainer is AWOL, the most likely outcome is that your program will be dropped from that release.
 
−
 
 
−
== How should it be packaged? ==
 
−
 
 
−
The key here is to minimize the maintenance load of the team building the disc. The idea is to make it quick and easy to update and rebuild the disc with a minimum of effort.
 
−
 
 
−
The best way to do this is to have your package already included in the main Ubuntu distribution. This should be your goal.
 
−
The next best thing is to have it packaged for Ubuntu and available via a Launchpad PPA (e.g. available from the UbuntuGIS repository). Next would be packaged into .debs from your own Ubuntu repository. An official Debian package could be used, but we'd have to rebuild it from source (this takes time but is fairly easy; usually these are already sucked in Ubuntu automatically so it isn't an issue).
 
−
 
 
−
* Programmers are encouraged to move their software into the [https://wiki.ubuntu.com/UbuntuGIS UbuntuGIS] or [http://wiki.debian.org/DebianGis DebianGIS] repositories.
 
−
 
 
−
If you must build from source, building a .deb package and installing that is preferred to "make install". If you do need to "make install", please set the path prefix to /usr/local/ and ''ensure that your package does not introduce namespace conflicts''.
 
−
 
 
−
In practice a number of scripts employ a hybrid approach, with as much as possible installed via official packages, and the occasional plugin compiled from source.
 
−
 
 
−
A base set of build-tools are already included, but if you install a bunch of -dev packages to build your app, make sure that you clean them all up after you are done.
 
−
 
 
−
== Which version to use? ==
 
−
 
 
−
* Stable, always! LiveGIS users are mainly starters. So they obviously are better with stable software. They have enough problems already and will gladly be happier without bleeding-edge software bugs.
 
−
 
 
−
For people who want the latest version, they can create a modified upgrade script.
 
−
 
 
−
In future, if space allows, in exceptional circumstances we will consider shipping beta software as a technology preview. These will have to remain a low priority in the battle for disc space, especially if a stable version of the software is already shipped on the disc.
 
−
 
 
−
= Directory Structure =
 
−
 
 
−
# '''/tmp''' Temporary files (e.g., downloaded archives) go into /tmp. Please create a separate folder for your project.
 
−
# '''/usr/lib''' application are usually installed into /usr/lib
 
−
# '''/usr/bin''' things that get executed by the user such as startup scripts or links to them should go into /usr/bin
 
−
# '''/etc/init.d''' startup/shutdown scripts for services (e.g., postgres, apache, tomcat) are stored in /etc/init.d
 
−
# '''/usr/local/share''' sample data and documentation goes into /usr/local/share
 
−
# '''/etc''' config files are stored in /etc
 
−
# '''/home/user''' user specific config files or working directories can go into <tt>/home/user/</tt>. However, keep files in /home/user as small as possible as this folder is loaded into memory in the Live DVD. Symlinks into /usr/local can be useful here.
 
−
 
 
−
 
 
−
* ''You are strongly encouraged to use '''/usr/local/''' and '''/var/local/''' for storing non-packaged content.''
 
−
 
 
−
= Mounting a Virtual Image =
 
−
For the purposes of faster dissemination of updates to the Live image via something like rsync you can mount an image file.
 
−
Make sure you turn off the virtual machine before you attempt to mount it, and that you unmount it before you attempt to run the vm again.
 
−
 
 
−
== Mount VMWare Server==
 
−
This method assumes you have vmware server installed, the key is that the vmware-mount script is on your system somewhere.
 
−
This instructions use /space/virtual as the vmware installation folder and /space/virtual/machines as the location of the disk images.
 
−
For more information see the [http://www.vmware.com/pdf/VMwareDiskMount.pdf VMware mounting guide]
 
−
*If it complains about not finding libdir/lib/libcrypto.so.0.9.8/libcrypto.so.0.9.8
 
−
*It's because it's looking for it in your vmware bin/libdir which doesn't exist so symlink
 
−
sudo ln -s /space/virtual/lib/vmware/ libdir
 
−
*Temporarily move your .vmx file out of the directory
 
−
*Move to you vmware bin folder
 
−
cd /space/virtual/bin/
 
−
sudo ./vmware-mount -p /space/virtual/machines/gisvm20090828x.vmdk
 
−
*Should be partition 1
 
−
*Make a directory to mount to somewhere easy to find, in my case where I keep my vm images
 
−
mkdir /space/virtual/machines/mountedimage
 
−
*Mount it
 
−
sudo ./vmware-mount /space/virtual/machines/gisvm20090828x.vmdk 1 /space/virtual/machines/mountedimage
 
−
*Don't forget to umount when you're done
 
−
sudo ./vmware-mount -d /space/virtual/machines/mountedimage
 
−
 
 
−
== Mount VirtualBox ==
 
−
These instructions have not been tested yet, they require VirtualBox 2.0+
 
−
http://forums.virtualbox.org/viewtopic.php?f=7&t=17574
 
−
 
 
−
== Run the the ISO in a QEMU / KVM Virtual Machine ==
 
−
(''Linux host only'')
 
−
 
 
−
If you have KVM installed use "<tt>qemu-kvm</tt>" or "<tt>kvm</tt>" instead of <tt>qemu</tt> on the command line. The usage is the same. It will be a ''lot'' faster. To run KVM you will have to have a newer CPU which supports hardware virtualization, a recent kernel, and have virtualization enabled in the BIOS. You'll want to make sure that the kvm kernel modules are loaded, /dev/kvm exists, and your user account belongs to the 'kvm' group. Older releases of QEMU may be partially accelerated by installing the KQEMU plugin. Without any sort of acceleration it will still run, but very very slowly.
 
−
 
 
−
=== Running live from a bootable ISO file ===
 
−
Specify that at least 512mb of RAM should be used. In this example 720mb has been requested which is a bit more comfortable and the ISO will be run live.
 
−
 
 
−
qemu -m 720 -cdrom osgeo_live.iso
 
−
or
 
−
kvm -m 720 -cdrom osgeo_live.iso
 
−
 
 
−
For me the mouse didn't work (it was just stuck in the middle of the screen). The fix was to set this environment variable before running <tt>kvm</tt>:
 
−
export SDL_VIDEO_X11_DGAMOUSE=0
 
−
 
 
−
* There is a good HowTo here: http://wiki.centos.org/HowTos/KVM
 
−
 
 
−
=== Install to a persistent VM ===
 
−
Create a permanent 5gb VM to work in:
 
−
qemu-img create -f qcow2 livedisc.img 6G
 
−
or
 
−
kvm-img create -f qcow2 livedisc.img 6G
 
−
 
 
−
Next start kvm with the ISO as the virtual CD drive and the VM image file as the the first hard drive (hd''a''). The "-boot d" option tells it to try to boot from the virtual CD drive  first.
 
−
: (from now on I'll just give the <tt>kvm</tt> version of the commands)
 
−
kvm -hda livedisc.img -cdrom osgeo_live.iso -m 720 -boot d
 
−
 
 
−
From within the VM, run the installer either from the boot menu or from the icon on the desktop of the Live OS. You should see the empty 6gb partition you created a few minutes ago. You can decide yourself if you want a swap space partition or not. I didn't, will adjust the -m size on the command line if I need more.
 
−
 
 
−
After it has installed you can restart qemu/kvm but without the -cdrom this time:
 
−
kvm -hda livedisc.img -m 720
 
−
 
 
−
At this time it might be a good idea to zip up a copy of the VM. If you want to start fresh later you can just delete the working image and replace it with the fresh version you zipped earlier.
 
−
 
 
−
=== Getting in via ssh ===
 
−
(ssh'ing out is trivial)
 
−
 
 
−
''Instructions on how to ssh into the QEMU/KVM VM can be found in the [http://wiki.laptop.org/go/Emulating_the_XO/Help_and_tips#SSH_into_qemu OLPC wiki].''
 
−
 
 
−
* Within the VM install the openssh-server package. This should generate RSA keys.
 
−
 
 
−
* Start the VM on the host with the command line option:
 
−
kvm [...] -redir 'tcp:2222::22'
 
−
 
 
−
* This will allow you to ssh to port 2222 on the host machine to get into the VM:
 
−
ssh -p 2222 user@127.0.0.1
 
−
 
 
−
* Add the "-nographic" command line option when you start up kvm if you want to start it headless.
 
−
* Add "-X" as a ssh command line option if you want to be able to tunnel X-Window apps from the VM.
 
−
 
 
−
== Mount the ISO as a readonly filesystem ==
 
−
:Not recommeded, it's preferred to get the image right to start, you will not be able to edit files inside the squashfs this way.
 
−
  mkdir test_iso
 
−
  mount -t iso9660 -o loop,ro livedvd.iso ./test_iso
 
−
 
 
−
If you want to do this without su rights, use the FuseISO package.
 
−
fuseiso -p livedvd.iso ./test_iso.$$
 
−
 
 
−
Unmount the FuseISO filesystem with:
 
−
fusermount -u ./test_iso.$$
 
−
=== Mount the ISO for changes ===
 
−
If you're on ubuntu to can edit and remaster and ISO with ISOMaster
 
−
sudo apt-get install isomaster
 
−
 
 
−
On Windows try Daemon tools and related software.
 
−
 
 
−
= Creating a fresh Virtual Machine =
 
−
== System Requirements ==
 
−
* RAM: You can only just get away with 1 Gig. 2 Gig is what you will require to avoid slowdowns due to memory swapping.
 
−
* About 10 Gig of spare hard disk for building a Virtual Machine.(20GB if you intend to build an ISO as well.)
 
−
* '''Recommended:''' An Ubuntu based linux distribution. We built this Live DVD process using an Ubuntu based system, and some of the steps assume the hosting operating system is Ubuntu, however you should be able to achieve all the steps (some slightly modified) from Windows.
 
−
 
 
−
== Selecting a Virtual Machine ==
 
−
 
 
−
The arramagong Live DVD can be built using a range of virtual machines. Here we explain our preferences and reasons:
 
−
 
 
−
; VirtualBox
 
−
: VirtualBox is provided as Open Source and can be downloaded onto an Ubuntu based system using apt-get which is great.
 
−
VirtualBox is our current recommendation, as it has been the more stable of our implementations so far.
 
−
 
 
−
Unfortunately, we haven't worked out how to convert from VirtualBox back to VMware images yet, and so far VMware Player software is more widely deployed than VirtualBox. So for the moment, we plan to build vmware images then convert to virtual box (rather than the other way around).
 
−
 
 
−
; VMware Server
 
−
: VMware server provides a browser based interface to the virtual machine which can be accessed over the internet. (This is a good way to let others view your virtual machine without them having to download the large Virtual Machine)
 
−
VMware Server also provides compression tools which we use at the end of the build process. (These are not provided by VMWare Player)
 
−
Unfortunately, there are instability issues with VMWare 2.0.2 + Ubuntu 9.04 / 9.10 (as a host), which means we have been unable to use this combination to build Arramagong 3.0. (The compression tools still seem to work).
 
−
The vmware images that are created can be used with VMWare Server, VMWare Player and relatively easily can be converted to VirtualBox format. (We haven't worked out how to convert from VirtualBox back to VMware images yet).
 
−
 
 
−
; VMware Player
 
−
: VMware Player provides a desktop application which runs a Virtual Machine, and can be used to create a vmware image. It doesn't provide the image compression utilities. (Can we download these from somewhere?)
 
−
 
 
−
; QEMU with KVM acceleration
 
−
: Very well supported by newer Linux kernels (KVM is the kernel developer's darling right now). Fully open source and installs from standard Ubuntu packages. Very easy to run. Only runs with Linux as the host, but can run any OS as the guest. See instructions elsewhere on this page. The kvm-img virtual-disk image creating program can create in a number of different VM formats by changing the -f parameter. (''used by HB'')
 
−
 
 
−
; Click2Try
 
−
: http:/click2try.com provide a web site for trialing virtual machines. They built an arramagong 2.0 release for FOSS4G 2009, and as of Feb 2009, have been working on setting an environment set up for 3.0. This should provide a good development environment once set up, but we will need to have a parallel VM development as well as I don't think they will not be building a VM for us.
 
−
 
 
−
== Install VirtualBox ==
 
−
 
 
−
 
 
−
== Install VMWare Server ==
 
−
<strike>
 
−
'''Note: During the building of Arramagong 3.0, we have found VMWare Server installed on Ubuntu 9.04 and 9.10 to be unstable to the point of being unusable.
 
−
 
 
−
We have successfully installed used a CentOS 5.3 host + VMware Server 2.0.2 (Make sure you don't accept any security updates as this introduces the bug which makes VMWare unstable). We suspect that Ubuntu 8.04 (LTS) + VMware should work as it is supposed to be supported, but we haven't verified this yet.
 
−
 
 
−
Alternatively, you can build images in VMWare Player.'''
 
−
 
 
−
You will need [http://www.vmware.com/products/server/ VMWare server] which provides all the tools for configuring a VM, setting the disk size, compressing the image etc. VirtualBox doesn't seem to have the compression scripts required, and it doesn't seem to have a way to copy your image to VMWare.
 
−
 
 
−
If you are installing on linux, the installer will ask a series of questions. Answer the default for everything, except the default administrator. For this, you need to enter in your <user_id> for the local computer you are installing on.
 
−
 
 
−
Once running, you should be able to access vmware server at:
 
−
https://127.0.0.1:8333
 
−
 
 
−
As explained [http://planetvm.net/blog/?p=1087 here], firefox has issues connecting to VM until you fix it by doing the following:
 
−
* Go to the url: about:config
 
−
* Update the following:
 
−
** security.enable.ssl2 from '''false''' to '''true'''
 
−
</strike>
 
−
 
 
−
== Create an Xubuntu Virtual Machine for VirtualBox ==
 
−
If you already have an Xubuntu (or similar) Virtual Machine, you can skip this step.
 
−
 
 
−
Otherwise, run VirtualBox, and create an image.
 
−
 
 
−
Download the latest  desktop i386 [http://www.xubuntu.org/ Xubuntu ISO]. Copy to the Virtual Machine directory:
 
−
mv xubuntu-10.04-desktop-i386.iso "~/.VirtualBox"
 
−
 
 
−
* Run VirtualBox
 
−
* Select Name=osgeolive
 
−
* Operating System=Linux
 
−
* Version=Ubuntu
 
−
* Base Memory Size (RAM) = 512 MB
 
−
* Create a new hard disk
 
−
* Select "Dynamically expanding storage"
 
−
* Hard Disk Size = 20 GB
 
−
To boot from the ISO:
 
−
* Select Settings->Storage->CD/DVD Device-> Add -> select the xubuntu ISO
 
−
* Select the Xubuntu ISO
 
−
To allow ssh access to the the VM:
 
−
* Select Settings -> Network -> Adapter 2 -> Enable Network Adapter -> Attached to: Bridged Adapter, Name: eth0
 
−
 
 
−
The following settings were required with an Ubuntu 9.10 guest operating system, but don't seem to be required for Ubuntu 10.04:
 
−
 
 
−
<strike>
 
−
If the display is not rendering for the VM, try:
 
−
* in Settings -> System -> Motherboard -> Extended Features:, untick "Enable ACPI"
 
−
* in Settings -> Display -> Video, set Video Memory = 64 MB.
 
−
</strike>
 
−
 
 
−
Run the Virtual Machine, which will boot from the Xubuntu CD. Select to install Xubuntu, using defaults, plus:
 
−
* Language: English (default)
 
−
* Location: (FOSS4G City)
 
−
* Prepare disk space: Erase and use the entire disk (default)
 
−
* Keyboard layout: USA (default)
 
−
* Name: user
 
−
* Login name: user
 
−
* Password: user
 
−
* Computer name: osgeolive
 
−
* [x] Log in automatically
 
−
 
 
−
After installing xubuntu, disconnect the ISO image from the CD and reference the physical drive instead.
 
−
* select Settings->Storage->CD/DVD Device: Empty
 
−
 
 
−
The save the virtual machine for use later on:
 
−
* select File->Export Appliance ... -> accept defaults
 
−
 
 
−
== Create an Xubuntu Virtual Machine for VMWare ==
 
−
<strike>If you already have an Xubuntu (or similar) Virtual Machine, you can skip this step.
 
−
 
 
−
Otherwise, run VMWare Server, and create an image.
 
−
 
 
−
Download the latest [http://www.xubuntu.org/ Xubuntu ISO]. Copy to the Virtual Machine directory:
 
−
 
 
−
wget http://mirror.internode.on.net/pub/ubuntu/xubuntu/9.04/release/xubuntu-9.04-desktop-i386.iso
 
−
mv xubuntu-9.04-desktop-i386.iso "/var/lib/vmware/Virtual Machines/"
 
−
 
 
−
Run vmware, which opens up a browser based User Interface.
 
−
vmware
 
−
 
 
−
Create a new Virtual Machine with:
 
−
* Virtual Machine name = "osgeolive"
 
−
* RAM = 512 Gig (we might want to experiment with less)
 
−
* Hard Disk = 20 Gig (If you plan to make an ISO you will need the space, just the VM install will fit in 8GB with little extra room, but it is a pain to increase later, and can't be done from VMWare Player)
 
−
* Mount the XUbuntu ISO image for the disk drive
 
−
 
 
−
For VirtualBox, select the virtual machine -> Settings
 
−
The following will allow ssh into the virtual machine (for easy building):
 
−
* Network -> Adapter 2 -> Enable Network Adapter -> Attached to: Bridged Adapter -> Name: eth0 (usually)
 
−
* Display -> Video Memory:64K, Enable 3D acceleration.
 
−
 
 
−
Run the Virtual Machine, which will boot from the Xubuntu CD. Select to install Xubuntu, using defaults, plus:
 
−
; What is your name?
 
−
: user
 
−
; What is your password?
 
−
: user
 
−
; What is the name of your computer?
 
−
: arramagong
 
−
; [x] Log in automatically
 
−
 
 
−
After installing xubuntu, in the VMWare Summary tab, select the change the CD drive from pointing to the xubuntu iso to point to the actual hard drive.
 
−
</strike>
 
−
== Alternative: build ISO using vmbuilder ==
 
−
<strike>
 
−
'''Side note:''' [https://help.ubuntu.com/community/JeOSVMBuilder vmbuilder] is reported to be a good way to create an Ubuntu image. I haven't tried it, and haven't worked out if it can create the smaller Xubuntu image. (Please update this wiki if you try it). Note: This method requires you to be running the same or newer version of Ubuntu than you want to make an image for.
 
−
 
 
−
host$ sudo apt-get install python-vm-builder
 
−
host$ sudo vmbuilder vmserver ubuntu --suite karmic --arch i386 \
 
−
      --mem 512 --rootsize=20000 --user user --pass user
 
−
</strike>
 
−
 
 
−
== Backup base Virtual Machine ==
 
−
You should now have a XUbuntu Virtual Machine image in /var/lib/vmware/Virtual Machines/ . It would be a good idea to save a copy of this machine somewhere, so you can use it again.
 
−
 
 
−
= Creating a Click2Try Virtual Machine =
 
−
[http://click2try.com Click2Try] provides a public website that people can use to create a Virtual Machine, and our project has been given a free account to use Click2Try as a development platform, which makes collaborative development much easier.
 
−
 
 
−
A'''s of 20 Feb 2010, the click2try website were having technical difficulties getting an Xunbutu 9.10 base image working. They are still working on it.'''
 
−
<strike>
 
−
== Click2Try Account details ==
 
−
Go to http://www.click2try.com and login with:
 
−
* User Name: osgeo
 
−
* password: .ask Cameron Shorter.
 
−
 
 
−
Once logged in, click on the “My Dashboard” link located on the upper right corner of the page.  Then click the “Application Manager” tab.  You will notice that there are two virtual machines already defined in your dashboard.  They are:
 
−
 
 
−
; base-xubuntu-9-04:
 
−
:This is a base xubuntu virtual machine.  Do not alter this one.  Create copies from this VM when you want a fresh virtual machine to start installing stuff on. If you have a point in time that you want to save.  Simply shut down the vm.  make a copy of it.  Rename the copy to what you want.
 
−
 
 
−
; base-arramagong-2-0:
 
−
: This is the base arramagong 2.0.3 application released at FOSS4G 2009. Do not run this version.
 
−
 
 
−
; arramagong-2-0:
 
−
: This is the arramagong 2.0.3 application released at FOSS4G 2009. You can run this version.
 
−
 
 
−
 
 
−
== Create Click2Try base Xubuntu image ==
 
−
Tom Callighan from Click2Try created a base Xubuntu system for us. Ask him for a new Xubuntu base image, or ask him to fill in the steps so that we can do it.
 
−
 
 
−
== Click2Try: Add user=user ==
 
−
The base Xubuntu image only has a root user. The following needs to be done to add user="user", pass="user"
 
−
 
 
−
* In order to allow a low security password, Update /etc/common-password/common-password
 
−
 
 
−
Once we place above file then we can create user "user" and set password.
 
−
 
 
−
Here are the steps which we have followed.
 
−
 
 
−
 
 
−
#useradd user
 
−
 
 
−
#passwd user [Set the password as "user"]
 
−
 
 
−
#usermod -u 499 user
 
−
 
 
−
# su user
 
−
 
 
−
Above command will create directory once we login using user "user".
 
−
 
 
−
About ubuntu release version, yes for gisvm which we build  earlier stage was ubuntu904 as per requirement.
 
−
 
 
−
Please mail us if you find issues.
 
−
 
 
−
== Click2Try: Create an Arramagong base image ==
 
−
* Copy the base-xubuntu-9-10 image, call it arramagong-<version>
 
−
* Make the image external: Highlight the VM in the application manager, shut the VM down (make sure you save the session), click on the sharing tab,
 
−
click on the public access button.  This feature actually has a second benefit.  It keeps the VM running even when you leave the VM from the viewer.  That way, when you log back into click2try you can access the VM much quicker.
 
−
 
 
−
== Click2Try: Create default user ==
 
−
* Launch the new image you created
 
−
* Log in with: user=root, password=.Same as above.
 
−
</strike>
 
−
 
 
−
= Creating the GIS Virtual Machine =
 
−
 
 
−
== Set the Version Number and Changes ==
 
−
Update https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/VERSION.txt with the current version number.
 
−
 
 
−
Update https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/CHANGES.txt with changes since the last release.
 
−
 
 
−
== Copy base virtual machine ==
 
−
 
 
−
Copy the base virtual machine to a new directory and rename it to the current version:
 
−
 
 
−
host$ cd /var/lib/vmware/Virtual\ Machines/
 
−
host$ cp -pr xubuntu-9.10/ arramagong-3.0-rc2
 
−
 
 
−
* Log into the control panel: https://127.0.0.1:8333/ui
 
−
* Select the base xubuntu image VM: Inventory -> arramagong
 
−
* Select Summary tab
 
−
* In the "Commands" pane, select "Configure VM"
 
−
* Change the hostname from "arramagong" to a name which includes version number. Eg: "arramagong-3.0-rc2"
 
−
 
 
−
Alternative name change:
 
−
* You can change the image name in /var/lib/vmware/Virtual\ Machines/arramagong-3.0-rc2/arramagong.vmx . Update the following line as:
 
−
displayName = "arramagong-3.0rc2"
 
−
 
 
−
== Bootstrap the Live DVD ==
 
−
* Start the Xubuntu Virtual Machine
 
−
* Open a terminal
 
−
* Copy the bootstrap script into your home directory and execute it
 
−
arramagong$ cd /tmp
 
−
arramagong$ wget https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/bootstrap.sh
 
−
arramagong$ chmod a+x bootstrap.sh
 
−
arramagong$ sudo ./bootstrap.sh
 
−
* This will install subversion, and the install scripts, and create a link to them from your home directory.
 
−
 
 
−
== Install all applications ==
 
−
* '''Optional''': If you have run through this process a number of times, you might be smart enough to have saved a local copy of the tmp/ directory. You can save time and bandwidth by copying the tmp/ contents onto the live DVD:
 
−
sudo rsync -avz username@hostname.org:/path_to_tmp_dir/ /tmp/
 
−
sudo mkdir /var/cache/apt
 
−
sudo cp /tmp/apt/* /var/cache/apt/
 
−
* cd to the install scripts directory, and run the main.sh installer. Note, that we log the output into tee, so that we can search for errors later.
 
−
cd ~/gisvm/bin
 
−
sudo ./main.sh 2>&1 | tee /var/log/arramagong/main_install.log
 
−
* Answer prompts as they come up, there are a few at the start. Wait for a few hours while everything is downloaded and installed.
 
−
* When finished, and before you shutdown your virtual machine, copy the download directories in /tmp to your local computer. (This is to save you re-downloading the files if you are to run through the process again.)
 
−
* The Live DVD should now be ready to test.
 
−
 
 
−
= Virtual Machine specific steps =
 
−
Take a copy of the Virtual Machine, as you don't want VMWare tools installed on the ISO image (in the next step).
 
−
== Install VMWare tools ==
 
−
VMWare tools allow cut and paste from the host operating system to a VMWare virtual machine, and also speeds up graphic rendering on VMWare.
 
−
 
 
−
In the VMWare browser control panel, select the virtual machine, then "Summary" tab. Turn on the virtual machine, then select "Install VMWare tools". This mounts the vmware-tools scripts in the arramagong virtual machine, so that it can be installed. Which you can do using the following commands:
 
−
 
 
−
arramagong$ cd /tmp
 
−
arramagong$ tar -zxf /media/cdrom0/VMwareTools-7.7.5-156745.tar.gz
 
−
arramagong$ cd vmware-tools-distrib/
 
−
arramagong$ sudo ./vmware-install.pl
 
−
arramagong$ # answer defaults to all questions
 
−
 
 
−
== Power off ==
 
−
Then poweroff the virtual machine. (Note that this removes all files in the tmp/ directory).
 
−
arramagong$ sudo halt
 
−
 
 
−
== Package the VMWare VM ==
 
−
 
 
−
'''Note:''' ''Most of this step is covered in the package.sh script. Check and update the variables in the script before running on an Ubuntu based system.''
 
 
   
 
   
−
On the host computer, all temporally files can be removed from the virtual machine folder: log files, ram files, etc. Remove all files '''EXCEPT''': *'''.vmx''' (VM definition file) and *'''.vmdk''' (virtual disk file)
+
host$ VBoxManage clonehd osgeolive.vdi osgeolive-14.0-amd64.vmdk --format VMDK
−
 
+
OR with a recent version of QEMU
−
Shrink the virtual machine: (requires VMWare Server installed)
+
  host$ qemu-img convert -f vdi -o compat6 -O vmdk osgeolive.vdi osgeolive-14.0-amd64.vmdk
−
'''Note: If you get an error from this command, you may need to temporarily rename the *.vmx file before operation and then name it back after'''
 
−
 
 
−
  host$ vmware-vdiskmanager -k *.vmdk
 
  
 
Zip the image up:
 
Zip the image up:
−
  host$ 7z a -mx=9 ArramagongGISVM2009alpha4.7z ArramagongGISVM2009alpha4/
 
  
−
If the image is greater than 2 Gig, then you also need to split the image. The OSGeo download server isn't configured to accept files of a greater size.
+
  host$ 7z a -mx=9 osgeolive-14.0-amd64.vmdk.7z osgeolive-14.0-amd64.vmdk
−
  host$ split -b 1500M ArramagongGISVM2009alpha4.7z ArramagongGISVM2009alpha4.7z
 
  
 
Create the md5sum checksums, so which can be used to confirm that the images have been downloaded correctly:
 
Create the md5sum checksums, so which can be used to confirm that the images have been downloaded correctly:
 
  host$ md5sum *.7z*
 
  host$ md5sum *.7z*
  
−
= Build the Live DVD ISO image =
+
= Upload the Release =
  
−
== Build ISO ==
+
== Upload to sourceforge ==
−
The ISO will be built as part of the main.sh script and stored in /tmp/remastersys/ . Make sure to copy the ISO onto the osgeo server, and then remove /tmp/remastersys/ or reboot the VM to remove it(must boot up again) before packaging the VM.
+
As of 6.0 the official releases are hosted on sourceforge. To upload you need a sourceforge account and permissions to the osgeo-live project upload.
−
* Run the script to build the ISO, this includes downloading close to 1GB of Windows and Mac installers.
 
−
sudo ./build_iso.sh 2>&1 | tee /var/log/arramagong/build_iso.log
 
  
−
* Once the ISO is complete copy it out to a server (a local server is fastest)
+
  rsync -e ssh osgeolive-14.0-amd64.iso username,osgeo-live@frs.sourceforge.net:/home/pfs/project/o/os/osgeo-live/14.0/
−
  scp /tmp/remastersys/arramagong-livedvd-2.0-final.iso.md5 user@server.org:destination/path/
 
−
scp /tmp/remastersys/arramagong-livedvd-2.0-final.iso user@server.org:destination/path/
 
−
* And/OR Wget or scp the file to the upload.osgeo.org server (Note wget is much faster if you have a good webserver to host from)
 
  
−
== Backup files ==
+
== Upload to the OSGeo Server ==
−
'''Optional:''' To save time and bandwidth, it is a good idea to back up all the files downloaded into your /tmp directory and the apt-get cache (var/cache/apt), which you can then copy onto future images you create before starting the build process. These files are copied into /tmp/<version>/ during the build process:
 
−
host$ scp -pr user@<arramagong ip>/tmp/3.1alpha1 arragmagong_tmp_3.1alpha1
 
  
−
 
+
  host$ scp -pr osgeolive-gisvm-2.0-alpha5 username@upload.osgeo.org:/osgeo/download/livedvd/
−
= Upload to the OSGeo Server =
 
−
 
 
−
  host$ scp -pr arramagong-gisvm-2.0-alpha5 username@upload.osgeo.org:/osgeo/download/livedvd/
 
  
 
Update the index.html file at: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/download/index.html
 
Update the index.html file at: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/download/index.html
Line 604: Line 159:
 
Check the result at: http://download.osgeo.org/livedvd
 
Check the result at: http://download.osgeo.org/livedvd
  
−
=== Creating a torrent file ===
+
== Creating a torrent file ==
 +
'''Ibiblio has offered torrent hosting, this section needs to include who to contact/how to setup'''
 
Several tools exist for creating torrents. Below are the key settings you need.
 
Several tools exist for creating torrents. Below are the key settings you need.
  
Line 610: Line 166:
 
* Make sure you have a copy of the file you want to create a torrent for. Mounted remote drives may work.
 
* Make sure you have a copy of the file you want to create a torrent for. Mounted remote drives may work.
 
* Web Seed - URL to a web server that hosts the file, adds speed to the torrents especially at the start. (Most clients implement this now)
 
* Web Seed - URL to a web server that hosts the file, adds speed to the torrents especially at the start. (Most clients implement this now)
−
** Note: Currently OSGeo servers can not handle downloads of files over 2GB, do not web seed from OSGeo for larger files.
 
 
* Tracker - There are 2 open and free trackers that have been tested, use one or the other as most clients do not handle multiple trackers yet.
 
* Tracker - There are 2 open and free trackers that have been tested, use one or the other as most clients do not handle multiple trackers yet.
 
  http://tracker.openbittorrent.com/announce
 
  http://tracker.openbittorrent.com/announce
Line 619: Line 174:
 
* Upload the .torrent file of your iso file for others to grab
 
* Upload the .torrent file of your iso file for others to grab
 
* Start the torrent on your machine or dedicated seeder, so that others have somewhere to start.
 
* Start the torrent on your machine or dedicated seeder, so that others have somewhere to start.
 +
* There is a possibility of seeding from osgeo machines but a Quality of Service (QoS) system would need to be in place to ensure it does not impact other services.
  
 
= SVN branches and tags =
 
= SVN branches and tags =
Line 643: Line 199:
 
  cd -
 
  cd -
 
    
 
    
−
  svn copy branches/arramagong_2 tags/release_20090927_arramagong_2_0
+
  svn copy branches/arramagong_2 tags/release_20090927_arramagong_2_0_3
−
  svn commit tags/release_20090927_arramagong_2_0 -m "tag release 2.0"
+
  svn commit tags/release_20090927_arramagong_2_0_3 -m "tag release 2.0.3"
  
 
=== Merging ===
 
=== Merging ===
Line 667: Line 223:
  
 
[[Category: Live-demo]]
 
[[Category: Live-demo]]
 +
 +
 +
= Staging the new release =
 +
 +
== Website ==
 +
Log in to live.osgeo.org.
 +
cd /osgeo/osgeolive
 +
mkdir ''<versionnumber>''
 +
 +
From another machine (ususally adhoc) go to the root of the docs and do
 +
scp -r /osgeo/livedvd/www/docs/* live.osgeo.org:/osgeo/osgeolive/''<versionumber>''/
 +
 +
Once done check that it's all good:
 +
<nowiki>http://live.osgeo.org/archive/</nowiki>''versionnumber''
 +
 +
Assuming docs are ok, then update the current symlink:
 +
chmod -R g+w ''<versionnumber>''
 +
rm current
 +
ln -s ''<versionnumber>'' current
 +
 +
== Download mirror ==
 +
 +
Upload files to https://sourceforge.net/projects/osgeo-live/files/
 +
 +
Add a new folder for the release, go into it, then scp the file to SourceForge.
 +
* See  https://sourceforge.net/p/forge/documentation/Release%20Files%20for%20Download/#scp
 +
 +
Click the box to stage the files for 3 days to give time for the mirrors to populate.
 +
 +
Once the files are ready (no longer say "Pending...") and the download links and redirects from download.osgeo.org are tested & working, set the default download on the SourceForge site to either the full or mini ISO by clicking the "i" information button next to the file and "select all" for all platform types, then click the Save button.

Latest revision as of 08:02, 26 June 2022

Getting started

The first thing you will have to do is check-out the latest build scripts using either one of these two Git servers: 

git clone https://git.osgeo.org/gitea/osgeolive/OSGeoLive.git

git clone https://github.com/OSGeo/OSGeoLive.git
  • see here for more details.

How to add your project to OSGeo Live

Creating a fresh Virtual Machine to use as a build host (Optional)

Refer to: Live_GIS_Virtual_Machine.

Build the Live DVD ISO image

Build ISO

This section describes the new method for building OSGeoLive as described in official ubuntu wiki. This section is self-contained and there is no need to perform any of the procedures described above.

All you need is a running Ubuntu/Xubuntu/Kubuntu/Lubuntu installation (even within a virtual machine as long as it has ~20GB free disk space). All needed to be done are the following steps under a "user" account:

  • Bootstrap the host operating system. If you use the system to build more than once, then this must be done only for the first build
host$ cd /tmp
host$ wget https://github.com/OSGeo/OSGeoLive/raw/master/bin/bootstrap.sh
host$ chmod a+x bootstrap.sh
host$ sudo ./bootstrap.sh

This will install Git and the install scripts, and create a link to them from your home directory.

  • Set the Version Number and Changes
 Update https://github.com/OSGeo/OSGeoLive/blob/master/VERSION.txt with the current version number.

 Update https://github.com/OSGeo/OSGeoLive/blob/master/CHANGES.txt with changes since the last release.

This list can be a summary of the revision log between releases Commit the changes to Git through a Pull Request right before a release build.

  • Execute the build script:
host$ cd ~/gisvm/bin
host$ sudo ./build_chroot.sh amd64 release master OSGeo 2>&1 | tee /var/log/osgeolive/chroot-build.log
  • Compress the logs:
host$ cd ~/livecdtmp
host$ tar czf version-log.tar.gz -C /var/log osgeolive
  • After the completion of the above script the new iso file is located in ~/livecdtmp along with the build logs. In case you wish to rerun the build process, do not remove or move the lubuntu official iso located in this folder to skip downloading it again.
  • It is required to reboot your host machine after build is completed
  • Once the ISO is complete copy it out to a server (a local server is fastest)
scp ~/livecdtmp/osgeolive-mini-8.0.iso user@server.org:destination/path/
scp ~/livecdtmp/osgeolive-mini-8.0-log.tar.gz user@server.org:destination/path/
  • And/OR wget or scp the file to the upload.osgeo.org server (Note wget is much faster if you have a good webserver to host from)

Build Full ISO from Mini ISO

Any mini iso file resulting from the procedure below can be turned into full iso, which is identical but adds the Windows and Mac software installers. Note that the resulting file may be bigger than a DVD. In that case the list of included applications in the load_mac_installers.sh need to be edited to exclude additional applications.

The standard current rules used to pick applications:

  • Desktop Applications
  • OSGeo projects and Incubated projects
sudo ./build_full_iso.sh /full/path/to/osgeo-live-mini-8.0.iso 2>&1 | tee ~/build_full_iso.log

How to do development / debugging with the current build method

We have created a debug build process so that projects can now easily create their own iso, including parts of the OSGeoLive (eg only one project) in order to test if the installer scripts work well under this new build method. Here are the steps to debug/test your application:

One time steps

You will need to create a pure Lubuntu Virtual Machine setup:

  • Download lubuntu-22.04-desktop-amd64.iso from lubuntu web site.
  • Download and install VirtualBox.
  • Create a fresh VM installation of Lubuntu. You will need to create a virtual disk drive with at least 25 GB of space and allocate 768MB of RAM to the VM. During installation set the username to "user" and hostname to "osgeolive". DO NOT install system updates during lubuntu installation or after the installation is done. At this moment we work with the default kernel included in lubuntu.
  • After the VM is done, login as "user" and open a terminal.
  • Bootstrap the VM:
osgeolive$ cd /tmp
osgeolive$ wget https://github.com/OSGeo/OSGeoLive/raw/master/bin/bootstrap.sh
osgeolive$ chmod a+x bootstrap.sh
osgeolive$ sudo ./bootstrap.sh

This will install Git, the install scripts, and create a link to them from your home directory.

Steps to create the build (repeat as much as needed)

  • Make changes to your project's installation script and commit to Git.
  • Update the git code:
osgeolive$ cd ~/gisvm
osgeolive$ git pull origin master
  • Open file inchroot.sh with an editor and comment out all scripts you do not need for your test.
osgeolive$ cd ~/gisvm/bin
osgeolive$ vi inchroot.sh
  • Always leave un-commented the following scripts: setup.sh, install_services.sh, install_mysql.sh, install_java.sh, install_apache2.sh, install_tomcat6.sh, install_desktop.sh and setdown.sh.
  • Save your changes and execute the build:
osgeolive$ cd ~/gisvm/bin
osgeolive$ sudo ./build_chroot.sh amd64 2>&1 | tee /var/log/osgeolive/chroot-build.log
  • After a while the iso will be created in ~livecdtmp/
  • Do not delete the file ~/livecdtmp/lubuntu-22.04-desktop-amd64.iso as it will be needed for next build (saves time not to download again)
  • Logs are created at /var/log/osgeolive/chroot-build.log
  • Copy the iso and test

Build the Live DVD VM image

Create the VM

The OSGeoLive Virtual Machine creation process is now exactly similar to a plain Lubuntu VM installation. Use the mini iso file that was created from the previous chapter. Instructions can be found Live_GIS_Virtual_Machine

Package the VM

From within the VM, fill empty space with zeros in order to be able to shrink the virtual disk files: 

osgeolive$ sudo ~/gisvm/bin/zerofill.sh

Shrink the virtual machine: 

host$ VBoxManage modifyhd osgeolive.vdi --compact

Convert to vmdk format (more widely compatible): 

host$ VBoxManage clonehd osgeolive.vdi osgeolive-14.0-amd64.vmdk --format VMDK
OR with a recent version of QEMU
host$ qemu-img convert -f vdi -o compat6 -O vmdk osgeolive.vdi osgeolive-14.0-amd64.vmdk

Zip the image up: 

host$ 7z a -mx=9 osgeolive-14.0-amd64.vmdk.7z osgeolive-14.0-amd64.vmdk

Create the md5sum checksums, so which can be used to confirm that the images have been downloaded correctly: 

host$ md5sum *.7z*

Upload the Release

Upload to sourceforge

As of 6.0 the official releases are hosted on sourceforge. To upload you need a sourceforge account and permissions to the osgeo-live project upload. 

rsync -e ssh osgeolive-14.0-amd64.iso username,osgeo-live@frs.sourceforge.net:/home/pfs/project/o/os/osgeo-live/14.0/

Upload to the OSGeo Server

host$ scp -pr osgeolive-gisvm-2.0-alpha5 username@upload.osgeo.org:/osgeo/download/livedvd/

Update the index.html file at: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/download/index.html

Check the result at: http://download.osgeo.org/livedvd

Creating a torrent file

Ibiblio has offered torrent hosting, this section needs to include who to contact/how to setup Several tools exist for creating torrents. Below are the key settings you need.

Deluge with the torrent creator plugin has been tested.

  • Make sure you have a copy of the file you want to create a torrent for. Mounted remote drives may work.
  • Web Seed - URL to a web server that hosts the file, adds speed to the torrents especially at the start. (Most clients implement this now)
  • Tracker - There are 2 open and free trackers that have been tested, use one or the other as most clients do not handle multiple trackers yet.
http://tracker.openbittorrent.com/announce
udp://tracker.openbittorrent.com:80/announce

OR 

http://tracker.publicbt.com:80/announce
udp://tracker.publicbt.com:80/announce
  • Upload the .torrent file of your iso file for others to grab
  • Start the torrent on your machine or dedicated seeder, so that others have somewhere to start.
  • There is a possibility of seeding from osgeo machines but a Quality of Service (QoS) system would need to be in place to ensure it does not impact other services.

SVN branches and tags

Naming

  • main development happens in "trunk" (aka HEAD)
  • a branch is split off before release time. Bug fixes (only) happen in it and updates like version 2.1 are tagged from it.
its name might be like "arramagong_2"
  • a tag is a snapshot of the svn at some point in time, aka a release name. It does not make sense to checkin fixes to one. They should ideally be set as read-only after creation to prevent this.
its name might be like "release_20090927_arramagong_2_0"

Creating

cd livedvd/gisvm/

svn copy trunk branches/arramagong_2
svn commit branches/arramagong_2 -m "Splitting off branch for 2.x"

cd branches/arramagong_2/bin/
# something like this, maybe need a for loop or xargs:
sed -e 's+gisvm/trunk+gisvm/branches/arramagong_2+g' *.sh | less   # check
sed -i -e 's+gisvm/trunk+gisvm/branches/arramagong_2+g' *.sh       # execute
cd -
 
svn copy branches/arramagong_2 tags/release_20090927_arramagong_2_0_3
svn commit tags/release_20090927_arramagong_2_0_3 -m "tag release 2.0.3"

Merging

To merge a change from trunk into a release branch use "svn merge", as follows: (in this example r2131 from trunk) 

svn up branches/arramagong_2/
cd branches/arramagong_2/

svn merge -c 2131 https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk
svn diff
svn commit -m "bugfix: bikeshed should be orange! (merge from trunk r2131)"

If you do this a lot you might make a little shell script. Here's one called svn_merge_livefromtrunk.sh: 

#!/bin/sh
# Usage: svn_merge_livefromtrunk <rev number>
svn merge -c $1 https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk

See also


Staging the new release

Website

Log in to live.osgeo.org. 

cd /osgeo/osgeolive
mkdir <versionnumber>

From another machine (ususally adhoc) go to the root of the docs and do 

scp -r /osgeo/livedvd/www/docs/* live.osgeo.org:/osgeo/osgeolive/<versionumber>/

Once done check that it's all good: 

http://live.osgeo.org/archive/versionnumber

Assuming docs are ok, then update the current symlink: 

chmod -R g+w <versionnumber>
rm current
ln -s <versionnumber> current

Download mirror

Upload files to https://sourceforge.net/projects/osgeo-live/files/

Add a new folder for the release, go into it, then scp the file to SourceForge.

Click the box to stage the files for 3 days to give time for the mirrors to populate.

Once the files are ready (no longer say "Pending...") and the download links and redirects from download.osgeo.org are tested & working, set the default download on the SourceForge site to either the full or mini ISO by clicking the "i" information button next to the file and "select all" for all platform types, then click the Save button.

Retrieved from "http://wiki.osgeo.org/w/index.php?title=Live_GIS_Build&oldid=128684"