Difference between revisions of "Live GIS Build"

From OSGeo
Jump to navigation Jump to search
m (Modified Lubuntu OS version (18.04 => 22.04) and architecture (i386 => amd64))
 
(15 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 
= Getting started =
 
= Getting started =
  
The first thing you will have to do is check-out the latest build scripts using Subversion (SVN):
+
The first thing you will have to do is check-out the latest build scripts using either one of these two Git servers:
  svn checkout <nowiki>https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk</nowiki>
+
  git clone <nowiki>https://git.osgeo.org/gitea/osgeolive/OSGeoLive.git</nowiki>
  
* see [[Live_GIS_Disc#Subversion|here]] for more details.
+
git clone <nowiki>https://github.com/OSGeo/OSGeoLive.git</nowiki>
 +
 
 +
* see [[Live_GIS_Disc#Git_migration|here]] for more details.
  
 
=How to add your project to OSGeo Live=
 
=How to add your project to OSGeo Live=
Line 20: Line 22:
 
This section describes the new method for building OSGeoLive as described in [https://help.ubuntu.com/community/LiveCDCustomization official ubuntu wiki]. This section is self-contained and there is no need to perform any of the procedures described above.
 
This section describes the new method for building OSGeoLive as described in [https://help.ubuntu.com/community/LiveCDCustomization official ubuntu wiki]. This section is self-contained and there is no need to perform any of the procedures described above.
  
All you need is a running Ubuntu/Xubuntu/Kubuntu installation (even within a virtual machine as long as it has ~20GB free disk space). All needed to be done are the following steps under a "user" account:
+
All you need is a running Ubuntu/Xubuntu/Kubuntu/Lubuntu installation (even within a virtual machine as long as it has ~20GB free disk space). All needed to be done are the following steps under a "user" account:
  
 
* Bootstrap the host operating system. If you use the system to build more than once, then this must be done only for the first build  
 
* Bootstrap the host operating system. If you use the system to build more than once, then this must be done only for the first build  
 
  host$ cd /tmp
 
  host$ cd /tmp
  host$ wget https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/bootstrap.sh
+
  host$ wget https://github.com/OSGeo/OSGeoLive/raw/master/bin/bootstrap.sh
 
  host$ chmod a+x bootstrap.sh
 
  host$ chmod a+x bootstrap.sh
 
  host$ sudo ./bootstrap.sh
 
  host$ sudo ./bootstrap.sh
This will install subversion, and the install scripts, and create a link to them from your home directory.
+
This will install Git and the install scripts, and create a link to them from your home directory.
  
 
* Set the Version Number and Changes
 
* Set the Version Number and Changes
   Update https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/VERSION.txt with the current version number.
+
   Update https://github.com/OSGeo/OSGeoLive/blob/master/VERSION.txt with the current version number.
  
   Update https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/CHANGES.txt with changes since the last release.
+
   Update https://github.com/OSGeo/OSGeoLive/blob/master/CHANGES.txt with changes since the last release.
This list can be a summary of the [http://trac.osgeo.org/osgeo/log/livedvd/gisvm/trunk revision log] between releases
+
This list can be a summary of the [https://github.com/OSGeo/OSGeoLive/commits/master revision log] between releases
Commit the changes to svn right before a release build.
+
Commit the changes to Git through a Pull Request right before a release build.
  
 
* Execute the build script:
 
* Execute the build script:
 
  host$ cd ~/gisvm/bin
 
  host$ cd ~/gisvm/bin
host$ svn up
+
  host$ sudo ./build_chroot.sh amd64 release master OSGeo 2>&1 | tee /var/log/osgeolive/chroot-build.log
  host$ sudo ./build_chroot.sh 2>&1 | tee /var/log/osgeolive/chroot-build.log
 
  
 
* Compress the logs:
 
* Compress the logs:
Line 45: Line 46:
 
  host$ tar czf version-log.tar.gz -C /var/log osgeolive
 
  host$ tar czf version-log.tar.gz -C /var/log osgeolive
  
* After the completion of the above script the new iso file is located in ~/livecdtmp along with the build logs. In case you wish to rerun the build process, do not remove or move the xubuntu official iso located in this folder to skip downloading it again.
+
* After the completion of the above script the new iso file is located in ~/livecdtmp along with the build logs. In case you wish to rerun the build process, do not remove or move the lubuntu official iso located in this folder to skip downloading it again.
  
 
* It is required to reboot your host machine after build is completed
 
* It is required to reboot your host machine after build is completed
  
 
* Once the ISO is complete copy it out to a server (a local server is fastest)
 
* Once the ISO is complete copy it out to a server (a local server is fastest)
  scp ~/livecdtmp/osgeolive-mini-6.0.iso user@server.org:destination/path/
+
  scp ~/livecdtmp/osgeolive-mini-8.0.iso user@server.org:destination/path/
  scp ~/livecdtmp/osgeolive-mini-6.0-log.tar.gz user@server.org:destination/path/
+
  scp ~/livecdtmp/osgeolive-mini-8.0-log.tar.gz user@server.org:destination/path/
  
* And/OR Wget or scp the file to the upload.osgeo.org server (Note wget is much faster if you have a good webserver to host from)
+
* And/OR wget or scp the file to the upload.osgeo.org server (Note wget is much faster if you have a good webserver to host from)
  
 
=== Build Full ISO from Mini ISO ===
 
=== Build Full ISO from Mini ISO ===
Line 63: Line 64:
 
* OSGeo projects and Incubated projects
 
* OSGeo projects and Incubated projects
  
  sudo ./build_full_iso.sh /full/path/to/osgeo-live-mini-6.0.iso 2>&1 | tee ~/build_full_iso.log
+
  sudo ./build_full_iso.sh /full/path/to/osgeo-live-mini-8.0.iso 2>&1 | tee ~/build_full_iso.log
  
 
== How to do development / debugging with the current build method ==
 
== How to do development / debugging with the current build method ==
Line 74: Line 75:
 
You will need to create a pure Lubuntu Virtual Machine setup:
 
You will need to create a pure Lubuntu Virtual Machine setup:
  
* Download lubuntu-14.04-desktop-i386.iso from [http://cdimage.ubuntu.com/lubuntu/releases/14.04/release/lubuntu-14.04-desktop-i386.iso lubuntu web site].
+
* Download lubuntu-22.04-desktop-amd64.iso from [http://cdimage.ubuntu.com/lubuntu/releases/22.04/release/lubuntu-22.04-desktop-amd64.iso lubuntu web site].
 
* Download and install VirtualBox.
 
* Download and install VirtualBox.
 
* Create a fresh VM installation of Lubuntu. You will need to create a virtual disk drive with at least 25 GB of space and allocate 768MB of RAM to the VM. During installation set the username to "user" and hostname to "osgeolive". DO NOT install system updates during lubuntu installation or after the installation is done. At this moment we work with the default kernel included in lubuntu.
 
* Create a fresh VM installation of Lubuntu. You will need to create a virtual disk drive with at least 25 GB of space and allocate 768MB of RAM to the VM. During installation set the username to "user" and hostname to "osgeolive". DO NOT install system updates during lubuntu installation or after the installation is done. At this moment we work with the default kernel included in lubuntu.
Line 80: Line 81:
 
* Bootstrap the VM:
 
* Bootstrap the VM:
 
  osgeolive$ cd /tmp
 
  osgeolive$ cd /tmp
  osgeolive$ wget https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/bootstrap.sh
+
  osgeolive$ wget https://github.com/OSGeo/OSGeoLive/raw/master/bin/bootstrap.sh
 
  osgeolive$ chmod a+x bootstrap.sh
 
  osgeolive$ chmod a+x bootstrap.sh
 
  osgeolive$ sudo ./bootstrap.sh
 
  osgeolive$ sudo ./bootstrap.sh
  
This will install subversion, and the install scripts, and create a link to them from your home directory.
+
This will install Git, the install scripts, and create a link to them from your home directory.
  
 
===Steps to create the build (repeat as much as needed)===
 
===Steps to create the build (repeat as much as needed)===
  
* Make changes to your project's installation script and commit to svn.
+
* Make changes to your project's installation script and commit to Git.
  
* Update the subversion code:
+
* Update the git code:
 
  osgeolive$ cd ~/gisvm
 
  osgeolive$ cd ~/gisvm
  osgeolive$ svn up
+
  osgeolive$ git pull origin master
  
* Open file inchroot_debug.sh with an editor and comment out all scripts you do not need for your test.  
+
* Open file inchroot.sh with an editor and comment out all scripts you do not need for your test.  
 
  osgeolive$ cd ~/gisvm/bin
 
  osgeolive$ cd ~/gisvm/bin
  osgeolive$ vim inchroot_nightly.sh
+
  osgeolive$ vi inchroot.sh
  
* Always leave un-commented the following scripts: setup.sh, install_services.sh, install_mysql.sh, install_java.sh, install_apache2.sh, install_tomcat6.sh, install_desktop.sh and setdown.sh. We have already commented out all project installation scripts (except the needed ones) for you.
+
* Always leave un-commented the following scripts: setup.sh, install_services.sh, install_mysql.sh, install_java.sh, install_apache2.sh, install_tomcat6.sh, install_desktop.sh and setdown.sh.  
  
 
* Save your changes and execute the build:
 
* Save your changes and execute the build:
 
  osgeolive$ cd ~/gisvm/bin
 
  osgeolive$ cd ~/gisvm/bin
  osgeolive$ sudo ./build_chroot_nightly.sh i386 2>&1 | tee /var/log/osgeolive/chroot-build.log
+
  osgeolive$ sudo ./build_chroot.sh amd64 2>&1 | tee /var/log/osgeolive/chroot-build.log
  
 
* After a while the iso will be created in ~livecdtmp/
 
* After a while the iso will be created in ~livecdtmp/
  
* Do not delete the file ~livecdtmp/lubuntu-14.04-desktop-i386.iso as it will be needed for next build (saves time not to download again)
+
* Do not delete the file ~/livecdtmp/lubuntu-22.04-desktop-amd64.iso as it will be needed for next build (saves time not to download again)
  
 
* Logs are created at /var/log/osgeolive/chroot-build.log
 
* Logs are created at /var/log/osgeolive/chroot-build.log
Line 128: Line 129:
 
Shrink the virtual machine:  
 
Shrink the virtual machine:  
  
  host$ VBoxManage modifyhd osgeolive-vm-6.0/osgeolive.vdi --compact
+
  host$ VBoxManage modifyhd osgeolive.vdi --compact
  
 
Convert to vmdk format (more widely compatible):
 
Convert to vmdk format (more widely compatible):
 
   
 
   
  host$ VBoxManage clonehd osgeolive-vm-6.0/osgeolive.vdi osgeolive-vm-6.0/osgeo-live-6.0.vmdk --format VMDK
+
  host$ VBoxManage clonehd osgeolive.vdi osgeolive-14.0-amd64.vmdk --format VMDK
 
  OR with a recent version of QEMU
 
  OR with a recent version of QEMU
  host$ qemu-img convert -f vdi -o compat6 -O vmdk osgeolive-vm-6.0/osgeolive.vdi osgeolive-vm-6.0/osgeo-live-6.0.vmdk
+
  host$ qemu-img convert -f vdi -o compat6 -O vmdk osgeolive.vdi osgeolive-14.0-amd64.vmdk
  
 
Zip the image up:
 
Zip the image up:
  
  host$ 7z a -mx=9 osgeolive-vm-6.0.7z osgeolive-vm-6.0/osgeo-live-6.0.vmdk
+
  host$ 7z a -mx=9 osgeolive-14.0-amd64.vmdk.7z osgeolive-14.0-amd64.vmdk
  
 
Create the md5sum checksums, so which can be used to confirm that the images have been downloaded correctly:
 
Create the md5sum checksums, so which can be used to confirm that the images have been downloaded correctly:
Line 148: Line 149:
 
As of 6.0 the official releases are hosted on sourceforge. To upload you need a sourceforge account and permissions to the osgeo-live project upload.
 
As of 6.0 the official releases are hosted on sourceforge. To upload you need a sourceforge account and permissions to the osgeo-live project upload.
  
  rsync -e ssh osgeo-live-5.5.iso username,osgeo-live@frs.sourceforge.net:/home/pfs/project/o/os/osgeo-live/5.5/
+
  rsync -e ssh osgeolive-14.0-amd64.iso username,osgeo-live@frs.sourceforge.net:/home/pfs/project/o/os/osgeo-live/14.0/
  
 
== Upload to the OSGeo Server ==
 
== Upload to the OSGeo Server ==
Line 232: Line 233:
  
 
From another machine (ususally adhoc) go to the root of the docs and do
 
From another machine (ususally adhoc) go to the root of the docs and do
  scp -r * live.osgeo.org:/osgeo/osgeolive/''<versionumber>''/
+
  scp -r /osgeo/livedvd/www/docs/* live.osgeo.org:/osgeo/osgeolive/''<versionumber>''/
  
 
Once done check that it's all good:
 
Once done check that it's all good:
 
  <nowiki>http://live.osgeo.org/archive/</nowiki>''versionnumber''
 
  <nowiki>http://live.osgeo.org/archive/</nowiki>''versionnumber''
  
If it is then update the current symlink:
+
Assuming docs are ok, then update the current symlink:
 
  chmod -R g+w ''<versionnumber>''
 
  chmod -R g+w ''<versionnumber>''
 +
rm current
 
  ln -s ''<versionnumber>'' current
 
  ln -s ''<versionnumber>'' current
  

Latest revision as of 09:02, 26 June 2022

Getting started

The first thing you will have to do is check-out the latest build scripts using either one of these two Git servers:

git clone https://git.osgeo.org/gitea/osgeolive/OSGeoLive.git
git clone https://github.com/OSGeo/OSGeoLive.git
  • see here for more details.

How to add your project to OSGeo Live

Creating a fresh Virtual Machine to use as a build host (Optional)

Refer to: Live_GIS_Virtual_Machine.

Build the Live DVD ISO image

Build ISO

This section describes the new method for building OSGeoLive as described in official ubuntu wiki. This section is self-contained and there is no need to perform any of the procedures described above.

All you need is a running Ubuntu/Xubuntu/Kubuntu/Lubuntu installation (even within a virtual machine as long as it has ~20GB free disk space). All needed to be done are the following steps under a "user" account:

  • Bootstrap the host operating system. If you use the system to build more than once, then this must be done only for the first build
host$ cd /tmp
host$ wget https://github.com/OSGeo/OSGeoLive/raw/master/bin/bootstrap.sh
host$ chmod a+x bootstrap.sh
host$ sudo ./bootstrap.sh

This will install Git and the install scripts, and create a link to them from your home directory.

  • Set the Version Number and Changes
 Update https://github.com/OSGeo/OSGeoLive/blob/master/VERSION.txt with the current version number.
 Update https://github.com/OSGeo/OSGeoLive/blob/master/CHANGES.txt with changes since the last release.

This list can be a summary of the revision log between releases Commit the changes to Git through a Pull Request right before a release build.

  • Execute the build script:
host$ cd ~/gisvm/bin
host$ sudo ./build_chroot.sh amd64 release master OSGeo 2>&1 | tee /var/log/osgeolive/chroot-build.log
  • Compress the logs:
host$ cd ~/livecdtmp
host$ tar czf version-log.tar.gz -C /var/log osgeolive
  • After the completion of the above script the new iso file is located in ~/livecdtmp along with the build logs. In case you wish to rerun the build process, do not remove or move the lubuntu official iso located in this folder to skip downloading it again.
  • It is required to reboot your host machine after build is completed
  • Once the ISO is complete copy it out to a server (a local server is fastest)
scp ~/livecdtmp/osgeolive-mini-8.0.iso user@server.org:destination/path/
scp ~/livecdtmp/osgeolive-mini-8.0-log.tar.gz user@server.org:destination/path/
  • And/OR wget or scp the file to the upload.osgeo.org server (Note wget is much faster if you have a good webserver to host from)

Build Full ISO from Mini ISO

Any mini iso file resulting from the procedure below can be turned into full iso, which is identical but adds the Windows and Mac software installers. Note that the resulting file may be bigger than a DVD. In that case the list of included applications in the load_mac_installers.sh need to be edited to exclude additional applications.

The standard current rules used to pick applications:

  • Desktop Applications
  • OSGeo projects and Incubated projects
sudo ./build_full_iso.sh /full/path/to/osgeo-live-mini-8.0.iso 2>&1 | tee ~/build_full_iso.log

How to do development / debugging with the current build method

We have created a debug build process so that projects can now easily create their own iso, including parts of the OSGeoLive (eg only one project) in order to test if the installer scripts work well under this new build method. Here are the steps to debug/test your application:

One time steps

You will need to create a pure Lubuntu Virtual Machine setup:

  • Download lubuntu-22.04-desktop-amd64.iso from lubuntu web site.
  • Download and install VirtualBox.
  • Create a fresh VM installation of Lubuntu. You will need to create a virtual disk drive with at least 25 GB of space and allocate 768MB of RAM to the VM. During installation set the username to "user" and hostname to "osgeolive". DO NOT install system updates during lubuntu installation or after the installation is done. At this moment we work with the default kernel included in lubuntu.
  • After the VM is done, login as "user" and open a terminal.
  • Bootstrap the VM:
osgeolive$ cd /tmp
osgeolive$ wget https://github.com/OSGeo/OSGeoLive/raw/master/bin/bootstrap.sh
osgeolive$ chmod a+x bootstrap.sh
osgeolive$ sudo ./bootstrap.sh

This will install Git, the install scripts, and create a link to them from your home directory.

Steps to create the build (repeat as much as needed)

  • Make changes to your project's installation script and commit to Git.
  • Update the git code:
osgeolive$ cd ~/gisvm
osgeolive$ git pull origin master
  • Open file inchroot.sh with an editor and comment out all scripts you do not need for your test.
osgeolive$ cd ~/gisvm/bin
osgeolive$ vi inchroot.sh
  • Always leave un-commented the following scripts: setup.sh, install_services.sh, install_mysql.sh, install_java.sh, install_apache2.sh, install_tomcat6.sh, install_desktop.sh and setdown.sh.
  • Save your changes and execute the build:
osgeolive$ cd ~/gisvm/bin
osgeolive$ sudo ./build_chroot.sh amd64 2>&1 | tee /var/log/osgeolive/chroot-build.log
  • After a while the iso will be created in ~livecdtmp/
  • Do not delete the file ~/livecdtmp/lubuntu-22.04-desktop-amd64.iso as it will be needed for next build (saves time not to download again)
  • Logs are created at /var/log/osgeolive/chroot-build.log
  • Copy the iso and test

Build the Live DVD VM image

Create the VM

The OSGeoLive Virtual Machine creation process is now exactly similar to a plain Lubuntu VM installation. Use the mini iso file that was created from the previous chapter. Instructions can be found Live_GIS_Virtual_Machine

Package the VM

From within the VM, fill empty space with zeros in order to be able to shrink the virtual disk files:

osgeolive$ sudo ~/gisvm/bin/zerofill.sh

Shrink the virtual machine:

host$ VBoxManage modifyhd osgeolive.vdi --compact

Convert to vmdk format (more widely compatible):

host$ VBoxManage clonehd osgeolive.vdi osgeolive-14.0-amd64.vmdk --format VMDK
OR with a recent version of QEMU
host$ qemu-img convert -f vdi -o compat6 -O vmdk osgeolive.vdi osgeolive-14.0-amd64.vmdk

Zip the image up:

host$ 7z a -mx=9 osgeolive-14.0-amd64.vmdk.7z osgeolive-14.0-amd64.vmdk

Create the md5sum checksums, so which can be used to confirm that the images have been downloaded correctly:

host$ md5sum *.7z*

Upload the Release

Upload to sourceforge

As of 6.0 the official releases are hosted on sourceforge. To upload you need a sourceforge account and permissions to the osgeo-live project upload.

rsync -e ssh osgeolive-14.0-amd64.iso username,osgeo-live@frs.sourceforge.net:/home/pfs/project/o/os/osgeo-live/14.0/

Upload to the OSGeo Server

host$ scp -pr osgeolive-gisvm-2.0-alpha5 username@upload.osgeo.org:/osgeo/download/livedvd/

Update the index.html file at: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/download/index.html

Check the result at: http://download.osgeo.org/livedvd

Creating a torrent file

Ibiblio has offered torrent hosting, this section needs to include who to contact/how to setup Several tools exist for creating torrents. Below are the key settings you need.

Deluge with the torrent creator plugin has been tested.

  • Make sure you have a copy of the file you want to create a torrent for. Mounted remote drives may work.
  • Web Seed - URL to a web server that hosts the file, adds speed to the torrents especially at the start. (Most clients implement this now)
  • Tracker - There are 2 open and free trackers that have been tested, use one or the other as most clients do not handle multiple trackers yet.
http://tracker.openbittorrent.com/announce
udp://tracker.openbittorrent.com:80/announce

OR

http://tracker.publicbt.com:80/announce
udp://tracker.publicbt.com:80/announce
  • Upload the .torrent file of your iso file for others to grab
  • Start the torrent on your machine or dedicated seeder, so that others have somewhere to start.
  • There is a possibility of seeding from osgeo machines but a Quality of Service (QoS) system would need to be in place to ensure it does not impact other services.

SVN branches and tags

Naming

  • main development happens in "trunk" (aka HEAD)
  • a branch is split off before release time. Bug fixes (only) happen in it and updates like version 2.1 are tagged from it.
its name might be like "arramagong_2"
  • a tag is a snapshot of the svn at some point in time, aka a release name. It does not make sense to checkin fixes to one. They should ideally be set as read-only after creation to prevent this.
its name might be like "release_20090927_arramagong_2_0"

Creating

cd livedvd/gisvm/

svn copy trunk branches/arramagong_2
svn commit branches/arramagong_2 -m "Splitting off branch for 2.x"

cd branches/arramagong_2/bin/
# something like this, maybe need a for loop or xargs:
sed -e 's+gisvm/trunk+gisvm/branches/arramagong_2+g' *.sh | less   # check
sed -i -e 's+gisvm/trunk+gisvm/branches/arramagong_2+g' *.sh       # execute
cd -
 
svn copy branches/arramagong_2 tags/release_20090927_arramagong_2_0_3
svn commit tags/release_20090927_arramagong_2_0_3 -m "tag release 2.0.3"

Merging

To merge a change from trunk into a release branch use "svn merge", as follows: (in this example r2131 from trunk)

svn up branches/arramagong_2/
cd branches/arramagong_2/

svn merge -c 2131 https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk
svn diff
svn commit -m "bugfix: bikeshed should be orange! (merge from trunk r2131)"

If you do this a lot you might make a little shell script. Here's one called svn_merge_livefromtrunk.sh:

#!/bin/sh
# Usage: svn_merge_livefromtrunk <rev number>
svn merge -c $1 https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk

See also


Staging the new release

Website

Log in to live.osgeo.org.

cd /osgeo/osgeolive
mkdir <versionnumber>

From another machine (ususally adhoc) go to the root of the docs and do

scp -r /osgeo/livedvd/www/docs/* live.osgeo.org:/osgeo/osgeolive/<versionumber>/

Once done check that it's all good:

http://live.osgeo.org/archive/versionnumber

Assuming docs are ok, then update the current symlink:

chmod -R g+w <versionnumber>
rm current
ln -s <versionnumber> current

Download mirror

Upload files to https://sourceforge.net/projects/osgeo-live/files/

Add a new folder for the release, go into it, then scp the file to SourceForge.

Click the box to stage the files for 3 days to give time for the mirrors to populate.

Once the files are ready (no longer say "Pending...") and the download links and redirects from download.osgeo.org are tested & working, set the default download on the SourceForge site to either the full or mini ISO by clicking the "i" information button next to the file and "select all" for all platform types, then click the Save button.