Difference between revisions of "Live GIS Build"

From OSGeo
Jump to navigation Jump to search
(47 intermediate revisions by 6 users not shown)
Line 1: Line 1:
 
= Getting started =
 
= Getting started =
  
−
The first thing you will have to do is check-out the latest build scripts using Subversion (SVN):
+
The first thing you will have to do is check-out the latest build scripts using either one of these two Git servers:
−
  svn checkout <nowiki>https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk</nowiki>
+
  git clone <nowiki>https://git.osgeo.org/gitea/osgeolive/OSGeoLive.git</nowiki>
  
−
* see [[Live_GIS_Disc#Subversion|here]] for more details.
+
git clone <nowiki>https://github.com/OSGeo/OSGeoLive.git</nowiki>
 +
 
 +
* see [[Live_GIS_Disc#Git_migration|here]] for more details.
  
 
=How to add your project to OSGeo Live=
 
=How to add your project to OSGeo Live=
Line 10: Line 12:
 
* [[Live_GIS_Add_Project|Instructions for adding new projects]]
 
* [[Live_GIS_Add_Project|Instructions for adding new projects]]
  
−
= Mounting a Virtual Image =
+
= Creating a fresh Virtual Machine to use as a build host (Optional) =
−
For the purposes of faster dissemination of updates to the Live image via something like rsync you can mount an image file.
+
Refer to: [[Live_GIS_Virtual_Machine]].
−
Make sure you turn off the virtual machine before you attempt to mount it, and that you unmount it before you attempt to run the vm again.
 
  
−
== Mount VMWare Server==
+
= Build the Live DVD ISO image =
−
This method assumes you have vmware server installed, the key is that the vmware-mount script is on your system somewhere.
 
−
This instructions use /space/virtual as the vmware installation folder and /space/virtual/machines as the location of the disk images.
 
−
For more information see the [http://www.vmware.com/pdf/VMwareDiskMount.pdf VMware mounting guide]
 
−
*If it complains about not finding libdir/lib/libcrypto.so.0.9.8/libcrypto.so.0.9.8
 
−
*It's because it's looking for it in your vmware bin/libdir which doesn't exist so symlink
 
−
sudo ln -s /space/virtual/lib/vmware/ libdir
 
−
*Temporarily move your .vmx file out of the directory
 
−
*Move to you vmware bin folder
 
−
cd /space/virtual/bin/
 
−
sudo ./vmware-mount -p /space/virtual/machines/gisvm20090828x.vmdk
 
−
*Should be partition 1
 
−
*Make a directory to mount to somewhere easy to find, in my case where I keep my vm images
 
−
mkdir /space/virtual/machines/mountedimage
 
−
*Mount it
 
−
sudo ./vmware-mount /space/virtual/machines/gisvm20090828x.vmdk 1 /space/virtual/machines/mountedimage
 
−
*Don't forget to umount when you're done
 
−
sudo ./vmware-mount -d /space/virtual/machines/mountedimage
 
  
−
== Mount VirtualBox ==
 
−
These instructions have not been tested yet, they require VirtualBox 2.0+
 
−
http://forums.virtualbox.org/viewtopic.php?f=7&t=17574
 
  
−
== Run the the ISO in a QEMU / KVM Virtual Machine ==
+
== Build ISO ==
−
(''Linux host only'')
 
  
−
If you have KVM installed use "<tt>qemu-kvm</tt>" or "<tt>kvm</tt>" instead of <tt>qemu</tt> on the command line. The usage is the same. It will be a ''lot'' faster. To run KVM you will have to have a newer CPU which supports hardware virtualization, a recent kernel, and have virtualization enabled in the BIOS. You'll want to make sure that the kvm kernel modules are loaded, /dev/kvm exists, and your user account belongs to the 'kvm' group. Older releases of QEMU may be partially accelerated by installing the KQEMU plugin. Without any sort of acceleration it will still run, but very very slowly.
+
This section describes the new method for building OSGeoLive as described in [https://help.ubuntu.com/community/LiveCDCustomization official ubuntu wiki]. This section is self-contained and there is no need to perform any of the procedures described above.
  
−
=== Running live from a bootable ISO file ===
+
All you need is a running Ubuntu/Xubuntu/Kubuntu/Lubuntu installation (even within a virtual machine as long as it has ~20GB free disk space). All needed to be done are the following steps under a "user" account:
−
Specify that at least 512mb of RAM should be used. In this example 720mb has been requested which is a bit more comfortable and the ISO will be run live.
 
  
−
  qemu -m 720 -cdrom osgeo_live.iso
+
* Bootstrap the host operating system. If you use the system to build more than once, then this must be done only for the first build
−
or
+
  host$ cd /tmp
−
  kvm -m 720 -cdrom osgeo_live.iso
+
host$ wget https://github.com/OSGeo/OSGeoLive/raw/master/bin/bootstrap.sh
 +
host$ chmod a+x bootstrap.sh
 +
  host$ sudo ./bootstrap.sh
 +
This will install Git and the install scripts, and create a link to them from your home directory.
  
−
For me the mouse didn't work (it was just stuck in the middle of the screen). The fix was to set this environment variable before running <tt>kvm</tt>:
+
* Set the Version Number and Changes
−
export SDL_VIDEO_X11_DGAMOUSE=0
+
  Update https://github.com/OSGeo/OSGeoLive/blob/master/VERSION.txt with the current version number.
  
−
* There is a good HowTo here: http://wiki.centos.org/HowTos/KVM
+
  Update https://github.com/OSGeo/OSGeoLive/blob/master/CHANGES.txt with changes since the last release.
 +
This list can be a summary of the [https://github.com/OSGeo/OSGeoLive/commits/master revision log] between releases
 +
Commit the changes to Git through a Pull Request right before a release build.
  
−
=== Install to a persistent VM ===
+
* Execute the build script:
−
Create a permanent 5gb VM to work in:
+
  host$ cd ~/gisvm/bin
−
  qemu-img create -f qcow2 livedisc.img 6G
+
  host$ sudo ./build_chroot.sh amd64 release master OSGeo 2>&1 | tee /var/log/osgeolive/chroot-build.log
−
or
 
−
  kvm-img create -f qcow2 livedisc.img 6G
 
  
−
Next start kvm with the ISO as the virtual CD drive and the VM image file as the the first hard drive (hd''a''). The "-boot d" option tells it to try to boot from the virtual CD drive first.
+
* Compress the logs:
−
: (from now on I'll just give the <tt>kvm</tt> version of the commands)
+
  host$ cd ~/livecdtmp
−
  kvm -hda livedisc.img -cdrom osgeo_live.iso -m 720 -boot d
+
  host$ tar czf version-log.tar.gz -C /var/log osgeolive
  
−
From within the VM, run the installer either from the boot menu or from the icon on the desktop of the Live OS. You should see the empty 6gb partition you created a few minutes ago. You can decide yourself if you want a swap space partition or not. I didn't, will adjust the -m size on the command line if I need more.
+
* After the completion of the above script the new iso file is located in ~/livecdtmp along with the build logs. In case you wish to rerun the build process, do not remove or move the lubuntu official iso located in this folder to skip downloading it again.
  
−
After it has installed you can restart qemu/kvm but without the -cdrom this time:
+
* It is required to reboot your host machine after build is completed
−
kvm -hda livedisc.img -m 720
 
  
−
At this time it might be a good idea to zip up a copy of the VM. If you want to start fresh later you can just delete the working image and replace it with the fresh version you zipped earlier.
+
* Once the ISO is complete copy it out to a server (a local server is fastest)
 +
scp ~/livecdtmp/osgeolive-mini-8.0.iso user@server.org:destination/path/
 +
scp ~/livecdtmp/osgeolive-mini-8.0-log.tar.gz user@server.org:destination/path/
  
−
=== Getting in via ssh ===
+
* And/OR wget or scp the file to the upload.osgeo.org server (Note wget is much faster if you have a good webserver to host from)
−
(ssh'ing out is trivial)
 
  
−
''Instructions on how to ssh into the QEMU/KVM VM can be found in the [http://wiki.laptop.org/go/Emulating_the_XO/Help_and_tips#SSH_into_qemu OLPC wiki].''
+
=== Build Full ISO from Mini ISO ===
  
−
* Within the VM install the openssh-server package. This should generate RSA keys.
+
Any mini iso file resulting from the procedure below can be turned into full iso, which is identical but adds the Windows and Mac software installers. Note that the resulting file may be bigger than a DVD. In that case the list of included applications in the load_mac_installers.sh need to be edited to exclude additional applications.  
  
−
* Start the VM on the host with the command line option:
+
The standard current rules used to pick applications:
−
kvm [...] -redir 'tcp:2222::22'
+
* Desktop Applications
 +
* OSGeo projects and Incubated projects
  
−
* This will allow you to ssh to port 2222 on the host machine to get into the VM:
+
sudo ./build_full_iso.sh /full/path/to/osgeo-live-mini-8.0.iso 2>&1 | tee ~/build_full_iso.log
−
ssh -p 2222 user@127.0.0.1
 
  
−
* Add the "-nographic" command line option when you start up kvm if you want to start it headless.
+
== How to do development / debugging with the current build method ==
−
* Add "-X" as a ssh command line option if you want to be able to tunnel X-Window apps from the VM.
 
  
−
== Mount the ISO as a readonly filesystem ==
+
We have created a debug build process so that projects can now easily create their own iso, including parts of the OSGeoLive (eg only one project) in order to test if the installer scripts work well under this new build method.
−
:Not recommeded, it's preferred to get the image right to start, you will not be able to edit files inside the squashfs this way.
+
Here are the steps to debug/test your application:
−
  mkdir test_iso
 
−
  mount -t iso9660 -o loop,ro livedvd.iso ./test_iso
 
  
−
If you want to do this without su rights, use the FuseISO package.
+
===One time steps===
−
fuseiso -p livedvd.iso ./test_iso.$$
 
  
−
Unmount the FuseISO filesystem with:
+
You will need to create a pure Lubuntu Virtual Machine setup:
−
fusermount -u ./test_iso.$$
 
−
=== Mount the ISO for changes ===
 
−
If you're on ubuntu to can edit and remaster and ISO with ISOMaster
 
−
sudo apt-get install isomaster
 
  
−
On Windows try Daemon tools and related software.
+
* Download lubuntu-18.04-desktop-i386.iso from [http://cdimage.ubuntu.com/lubuntu/releases/18.04/release/lubuntu-18.04-desktop-i386.iso lubuntu web site].
 +
* Download and install VirtualBox.
 +
* Create a fresh VM installation of Lubuntu. You will need to create a virtual disk drive with at least 25 GB of space and allocate 768MB of RAM to the VM. During installation set the username to "user" and hostname to "osgeolive". DO NOT install system updates during lubuntu installation or after the installation is done. At this moment we work with the default kernel included in lubuntu.
 +
* After the VM is done, login as "user" and open a terminal.
 +
* Bootstrap the VM:
 +
osgeolive$ cd /tmp
 +
osgeolive$ wget https://github.com/OSGeo/OSGeoLive/raw/master/bin/bootstrap.sh
 +
osgeolive$ chmod a+x bootstrap.sh
 +
osgeolive$ sudo ./bootstrap.sh
  
−
= Creating a fresh Virtual Machine =
+
This will install Git, the install scripts, and create a link to them from your home directory.
−
Refer to: [[Live_GIS_Virtual_Machine]].
 
  
−
= Creating the GIS Virtual Machine =
+
===Steps to create the build (repeat as much as needed)===
  
−
== Set the Version Number and Changes ==
+
* Make changes to your project's installation script and commit to Git.
−
Update https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/VERSION.txt with the current version number.
 
  
−
Update https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/CHANGES.txt with changes since the last release.
+
* Update the git code:
−
This list can be a summary of the [http://trac.osgeo.org/osgeo/log/livedvd/gisvm/trunk revision log] between releases
+
osgeolive$ cd ~/gisvm
 +
osgeolive$ git pull origin master
  
−
Commit the changes to svn right before a release build.
+
* Open file inchroot.sh with an editor and comment out all scripts you do not need for your test.
 +
osgeolive$ cd ~/gisvm/bin
 +
osgeolive$ vi inchroot.sh
  
−
== Copy base virtual machine ==
+
* Always leave un-commented the following scripts: setup.sh, install_services.sh, install_mysql.sh, install_java.sh, install_apache2.sh, install_tomcat6.sh, install_desktop.sh and setdown.sh.
  
−
Copy the base virtual machine to a new directory and rename it to the current version:  
+
* Save your changes and execute the build:
 +
osgeolive$ cd ~/gisvm/bin
 +
osgeolive$ sudo ./build_chroot.sh i386 2>&1 | tee /var/log/osgeolive/chroot-build.log
  
−
Use the [http://trac.osgeo.org/osgeo/browser/livedvd/gisvm/trunk/bin/renamevm.sh helper script renamevm.sh]
+
* After a while the iso will be created in ~livecdtmp/
−
host$ sudo ./renamevm.sh OSGeoLive-4.5base OSGeoLive-4.5rc1
 
  
−
* Log into the control panel: https://127.0.0.1:8333/ui
+
* Do not delete the file ~/livecdtmp/lubuntu-18.04-desktop-i386.iso as it will be needed for next build (saves time not to download again)
−
* Go to the menu Virtual Machine -> Add Virtual Machine to Inventory ->  and select your OSGeoLive-4.5rc1.vmx
 
−
* Your image can now be started. When prompted select "I copied it"
 
  
−
Or do it By hand
+
* Logs are created at /var/log/osgeolive/chroot-build.log
−
host$ cd /var/lib/vmware/Virtual\ Machines/
 
−
host$ cp -pr OSGeoLive-4.5base/ OSGeoLive-4.5rc1
 
  
−
* Log into the control panel: https://127.0.0.1:8333/ui
+
* Copy the iso and test
−
* Select the base xubuntu image VM: Inventory -> OSGeoLive-4.5rc1
 
−
* Select Summary tab
 
−
* In the "Commands" pane, select "Configure VM"
 
−
* Change the hostname from "arramagong" to a name which includes version number. Eg: "arramagong-3.0-rc2"
 
  
−
Alternative name change:
+
= Build the Live DVD VM image =
−
* You can change the image name in /var/lib/vmware/Virtual\ Machines/arramagong-3.0-rc2/arramagong.vmx . Update the following line as:
 
−
displayName = "arramagong-3.0rc2"
 
  
−
== Bootstrap the Live DVD ==
+
== Create the VM ==
−
* Start the Xubuntu Virtual Machine
 
−
* Open a terminal
 
−
* Copy the bootstrap script into your home directory and execute it
 
−
osgeolive$ cd /tmp
 
−
osgeolive$ wget https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/bootstrap.sh
 
−
osgeolive$ chmod a+x bootstrap.sh
 
−
osgeolive$ sudo ./bootstrap.sh
 
−
* This will install subversion, and the install scripts, and create a link to them from your home directory.
 
−
 
 
−
== Install all applications ==
 
−
'''Optional''': If you have run through this process a number of times, you might be smart enough to have saved a local copy of the tmp/ directory. You can save time and bandwidth by copying the tmp/ contents onto the live DVD
 
−
* Method 1
 
−
sudo rsync -avz username@hostname.org:/path_to_tmp_dir/ /tmp/
 
−
sudo mkdir /var/cache/apt
 
−
sudo cp /tmp/apt/* /var/cache/apt/
 
−
* Method 2
 
−
# Create a Virtual Disk with your Virtual Machine software (8-10GB should work).
 
−
# Attach the disk to your Virtual Machine, it will be something like /dev/sdb1 when you are in the Virtual Machine
 
−
# Mount the drive as the /tmp folder
 
−
sudo mount /dev/sdb1 /tmp
 
−
 
 
−
''Tip'': You may want to uninstall xscreensaver and reboot, as of Xubuntu 11.04 builds it occasionally causes the vm to freeze.
 
  
−
'''Required''':
+
The OSGeoLive Virtual Machine creation process is now exactly similar to a plain Lubuntu VM installation.
−
* Bring down the graphical user interface.
+
Use the mini iso file that was created from the previous chapter.  
−
sudo service lightdm stop
+
Instructions can be found [[Live_GIS_Virtual_Machine]]
−
* Change Directory (cd) to the install scripts directory, and run the main.sh installer. Note, that we log the output into tee, so that we can search for errors later.
 
−
cd ~/gisvm/bin
 
−
sudo ./main.sh 2>&1 | tee /var/log/osgeolive/main_install.log
 
−
* Answer prompts as they come up, there are a few at the start. Wait for a few hours while everything is downloaded and installed.
 
−
* Save the logs and upload them for viewing
 
−
cd
 
−
tar czf version-log.tar.gz -C /var/log osgeolive
 
−
scp version-log.tar.gz username@yourserver.com:gisvmlogs
 
−
* When finished, and before you shutdown your virtual machine, copy the download directories in /tmp to your local computer. (This is to save you re-downloading the files if you are to run through the process again.)
 
−
* Shutdown the virtual machine and if you used a second drive for tmp, remove it from the config.
 
−
* Boot the machine again and run a zero-ing operation to ensure it will compress.
 
−
sudo ./gisvm/bin/zerofill.sh
 
−
* Shutdown the machine again for packaging.
 
−
* The Live DVD should now be ready to test, however if you're going to package it do so before testing.
 
  
−
= Virtual Machine specific steps =
+
== Package the VM ==
−
Take a copy of the Virtual Machine, as you don't want VMWare tools installed on the ISO image (in the next step).
 
−
== Install VMWare tools ==
 
−
VMWare tools allow cut and paste from the host operating system to a VMWare virtual machine, and also speeds up graphic rendering on VMWare.
 
  
−
In the VMWare browser control panel, select the virtual machine, then "Summary" tab. Turn on the virtual machine, then select "Install VMWare tools". This mounts the vmware-tools scripts in the arramagong virtual machine, so that it can be installed. Which you can do using the following commands:
+
From within the VM, fill empty space with zeros in order to be able to shrink the virtual disk files:
  
−
  osgeolive$ cd /tmp
+
  osgeolive$ sudo ~/gisvm/bin/zerofill.sh
−
osgeolive$ tar -zxf /media/cdrom0/VMwareTools-7.7.5-156745.tar.gz
 
−
osgeolive$ cd vmware-tools-distrib/
 
−
osgeolive$ sudo ./vmware-install.pl
 
−
osgeolive$ # answer defaults to all questions
 
  
−
== Power off ==
+
Shrink the virtual machine:
−
Then poweroff the virtual machine. (Note that this removes all files in the tmp/ directory).
 
−
arramagong$ sudo halt
 
  
−
== Package the VMWare VM ==
+
host$ VBoxManage modifyhd osgeolive.vdi --compact
  
−
'''Note:''' ''Most of this step is covered in the package.sh script. Check and update the variables in the script before running on an Ubuntu based system.''
+
Convert to vmdk format (more widely compatible):
 
   
 
   
−
On the host computer, all temporally files can be removed from the virtual machine folder: log files, ram files, etc. Remove all files '''EXCEPT''': *'''.vmx''' (VM definition file) and *'''.vmdk''' (virtual disk file)
+
host$ VBoxManage clonehd osgeolive.vdi osgeolive-14.0-amd64.vmdk --format VMDK
−
 
+
OR with a recent version of QEMU
−
Shrink the virtual machine: (requires VMWare Server installed)
+
  host$ qemu-img convert -f vdi -o compat6 -O vmdk osgeolive.vdi osgeolive-14.0-amd64.vmdk
−
'''Note: If you get an error from this command, you may need to temporarily rename the *.vmx file before operation and then name it back after'''
 
−
 
 
−
  host$ vmware-vdiskmanager -k *.vmdk
 
  
 
Zip the image up:
 
Zip the image up:
−
  host$ 7z a -mx=9 ArramagongGISVM2009alpha4.7z ArramagongGISVM2009alpha4/
 
  
−
If the image is greater than 2 Gig, then you also need to split the image. The OSGeo download server isn't configured to accept files of a greater size.
+
  host$ 7z a -mx=9 osgeolive-14.0-amd64.vmdk.7z osgeolive-14.0-amd64.vmdk
−
  host$ split -b 1500M ArramagongGISVM2009alpha4.7z ArramagongGISVM2009alpha4.7z
 
  
 
Create the md5sum checksums, so which can be used to confirm that the images have been downloaded correctly:
 
Create the md5sum checksums, so which can be used to confirm that the images have been downloaded correctly:
 
  host$ md5sum *.7z*
 
  host$ md5sum *.7z*
  
−
= Build the Live DVD ISO image =
+
= Upload the Release =
−
 
 
−
== Build ISO ==
 
−
The ISO will be built as part of the main.sh script and stored in /tmp/remastersys/ . Make sure to copy the ISO onto the osgeo server, and then remove /tmp/remastersys/ or reboot the VM to remove it(must boot up again) before packaging the VM.
 
−
* Run the script to build the ISO without additional files.
 
−
sudo ./build_iso.sh mini 2>&1 | tee /var/log/osgeolive/build_iso-mini.log
 
  
−
* Run the script to build the ISO, this includes downloading close to 1GB of Windows and Mac installers.
+
== Upload to sourceforge ==
−
sudo ./build_iso.sh 2>&1 | tee /var/log/osgeolive/build_iso.log
+
As of 6.0 the official releases are hosted on sourceforge. To upload you need a sourceforge account and permissions to the osgeo-live project upload.
  
−
* Once the ISO is complete copy it out to a server (a local server is fastest)
+
  rsync -e ssh osgeolive-14.0-amd64.iso username,osgeo-live@frs.sourceforge.net:/home/pfs/project/o/os/osgeo-live/14.0/
−
  scp /tmp/remastersys/osgeolive-livedvd-2.0-final.iso.md5 user@server.org:destination/path/
 
−
scp /tmp/remastersys/osgeolive-livedvd-2.0-final.iso user@server.org:destination/path/
 
−
* And/OR Wget or scp the file to the upload.osgeo.org server (Note wget is much faster if you have a good webserver to host from)
 
  
−
== Backup files ==
+
== Upload to the OSGeo Server ==
−
'''Optional:''' To save time and bandwidth, it is a good idea to back up all the files downloaded into your /tmp directory and the apt-get cache (var/cache/apt), which you can then copy onto future images you create before starting the build process. These files are copied into /tmp/<version>/ during the build process:
 
−
host$ scp -pr user@<arramagong ip>/tmp/3.1alpha1 arragmagong_tmp_3.1alpha1
 
−
 
 
−
== Alternative Method To Build ISO ==
 
−
 
 
−
This section describes an alternative method for building OSGeoLive as described in [https://help.ubuntu.com/community/LiveCDCustomization official ubuntu wiki]. This section is self-contained and there is no need to perform any of the procedures described above.
 
−
 
 
−
All you need is a running Ubuntu/Xubuntu/Kubuntu installation (even within a virtual machine as long as it has ~20GB free disk space). All needed to be done are the following steps:
 
−
 
 
−
* Bootstrap the host operating system. If you use the system to build more than once, then this must be done only for the first build
 
−
host$ cd /tmp
 
−
host$ wget https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/bootstrap.sh
 
−
host$ chmod a+x bootstrap.sh
 
−
host$ sudo ./bootstrap.sh
 
−
This will install subversion, and the install scripts, and create a link to them from your home directory.
 
−
 
 
−
* Execute the build script:
 
−
host$ cd ~/gisvm/bin
 
−
host$ svn up
 
−
host$ sudo ./build_chroot.sh 2>&1 | tee /var/log/osgeolive/chroot-build.log
 
−
 
 
−
* Compress the logs:
 
−
host$ cd ~/livecdtmp
 
−
host$ tar czf version-log.tar.gz -C /var/log osgeolive
 
−
 
 
−
* After the completion of the above script the new iso file is located in ~/livecdtmp along with the build logs. In case you wish to rerun the build process, do not remove or move the xubuntu official iso located in this folder to skip downloading it again.
 
−
 
 
−
* It is required to reboot your host machine after build is completed
 
−
 
 
−
* You can now use the iso to simply install in a virtual machine in order to get the OSGeoLive VM version. The installation procedure is the same as a normal Ubuntu OS.
 
−
 
 
−
== How to do development / debugging with the Alternative build method ==
 
−
 
 
−
We have created a debug build process so that projects can now easily create their own iso, including parts of the OSGeoLive (eg only one project) in order to test if the installer scripts work well under this new build method.
 
−
Here are the steps to debug/test your application:
 
−
 
 
−
===One time steps===
 
−
 
 
−
You will need to create a pure Xubuntu Virtual Machine setup:
 
−
 
 
−
* Download xubuntu-12.04-desktop-i386.iso from [http://xubuntu.org/getxubuntu/ xubuntu web site].
 
−
* Download and install VirtualBox.
 
−
* Create a fresh VM installation of Xubuntu. You will need to create a virtual disk drive with at least 25 GB of space and allocate 768MB of RAM to the VM. During installation set the username to "user" and hostname to "osgeolive". DO NOT install system updates during xubuntu installation or after the installation is done. At this momment we work with the default kernel included in xubuntu.
 
−
* After the VM is done, login as "user" and open a terminal.
 
−
* Bootstrap the VM:
 
−
osgeolive$ cd /tmp
 
−
osgeolive$ wget https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/bin/bootstrap.sh
 
−
osgeolive$ chmod a+x bootstrap.sh
 
−
osgeolive$ sudo ./bootstrap.sh
 
−
 
 
−
This will install subversion, and the install scripts, and create a link to them from your home directory.
 
−
 
 
−
===Steps to create the build (repeat as much as needed)===
 
−
 
 
−
* Make changes to your project's installation script and commit to svn.
 
−
 
 
−
* Update the subversion code:
 
−
osgeolive$ cd ~/gisvm
 
−
osgeolive$ svn up
 
−
 
 
−
* Open file inchroot_debug.sh with an editor and comment out all scripts you do not need for your test.
 
−
osgeolive$ cd ~/gisvm/bin
 
−
osgeolive$ vim inchroot_debug.sh
 
−
 
 
−
* Always leave un-commented the following scripts: setup.sh, install_services.sh, install_mysql.sh, install_java.sh, install_apache2.sh, install_tomcat6.sh, install_desktop.sh and setdown.sh. We have already commented out all project installation scripts (except the needed ones) for you.
 
−
 
 
−
* Save your changes and execute the build:
 
−
osgeolive$ cd ~/gisvm/bin
 
−
osgeolive$ sudo ./build_chroot_debug.sh 2>&1 | tee /var/log/osgeolive/chroot-build.log
 
−
 
 
−
* After a while the iso will be created in ~livecdtmp/
 
−
 
 
−
* Copy the iso and test
 
−
 
 
−
= Upload to the OSGeo Server =
 
  
 
  host$ scp -pr osgeolive-gisvm-2.0-alpha5 username@upload.osgeo.org:/osgeo/download/livedvd/
 
  host$ scp -pr osgeolive-gisvm-2.0-alpha5 username@upload.osgeo.org:/osgeo/download/livedvd/
Line 318: Line 159:
 
Check the result at: http://download.osgeo.org/livedvd
 
Check the result at: http://download.osgeo.org/livedvd
  
−
=== Creating a torrent file ===
+
== Creating a torrent file ==
 +
'''Ibiblio has offered torrent hosting, this section needs to include who to contact/how to setup'''
 
Several tools exist for creating torrents. Below are the key settings you need.
 
Several tools exist for creating torrents. Below are the key settings you need.
  
Line 381: Line 223:
  
 
[[Category: Live-demo]]
 
[[Category: Live-demo]]
 +
 +
 +
= Staging the new release =
 +
 +
== Website ==
 +
Log in to live.osgeo.org.
 +
cd /osgeo/osgeolive
 +
mkdir ''<versionnumber>''
 +
 +
From another machine (ususally adhoc) go to the root of the docs and do
 +
scp -r /osgeo/livedvd/www/docs/* live.osgeo.org:/osgeo/osgeolive/''<versionumber>''/
 +
 +
Once done check that it's all good:
 +
<nowiki>http://live.osgeo.org/archive/</nowiki>''versionnumber''
 +
 +
Assuming docs are ok, then update the current symlink:
 +
chmod -R g+w ''<versionnumber>''
 +
rm current
 +
ln -s ''<versionnumber>'' current
 +
 +
== Download mirror ==
 +
 +
Upload files to https://sourceforge.net/projects/osgeo-live/files/
 +
 +
Add a new folder for the release, go into it, then scp the file to SourceForge.
 +
* See  https://sourceforge.net/p/forge/documentation/Release%20Files%20for%20Download/#scp
 +
 +
Click the box to stage the files for 3 days to give time for the mirrors to populate.
 +
 +
Once the files are ready (no longer say "Pending...") and the download links and redirects from download.osgeo.org are tested & working, set the default download on the SourceForge site to either the full or mini ISO by clicking the "i" information button next to the file and "select all" for all platform types, then click the Save button.

Revision as of 09:05, 22 May 2021

Getting started

The first thing you will have to do is check-out the latest build scripts using either one of these two Git servers: 

git clone https://git.osgeo.org/gitea/osgeolive/OSGeoLive.git

git clone https://github.com/OSGeo/OSGeoLive.git
  • see here for more details.

How to add your project to OSGeo Live

Creating a fresh Virtual Machine to use as a build host (Optional)

Refer to: Live_GIS_Virtual_Machine.

Build the Live DVD ISO image

Build ISO

This section describes the new method for building OSGeoLive as described in official ubuntu wiki. This section is self-contained and there is no need to perform any of the procedures described above.

All you need is a running Ubuntu/Xubuntu/Kubuntu/Lubuntu installation (even within a virtual machine as long as it has ~20GB free disk space). All needed to be done are the following steps under a "user" account:

  • Bootstrap the host operating system. If you use the system to build more than once, then this must be done only for the first build
host$ cd /tmp
host$ wget https://github.com/OSGeo/OSGeoLive/raw/master/bin/bootstrap.sh
host$ chmod a+x bootstrap.sh
host$ sudo ./bootstrap.sh

This will install Git and the install scripts, and create a link to them from your home directory.

  • Set the Version Number and Changes
 Update https://github.com/OSGeo/OSGeoLive/blob/master/VERSION.txt with the current version number.

 Update https://github.com/OSGeo/OSGeoLive/blob/master/CHANGES.txt with changes since the last release.

This list can be a summary of the revision log between releases Commit the changes to Git through a Pull Request right before a release build.

  • Execute the build script:
host$ cd ~/gisvm/bin
host$ sudo ./build_chroot.sh amd64 release master OSGeo 2>&1 | tee /var/log/osgeolive/chroot-build.log
  • Compress the logs:
host$ cd ~/livecdtmp
host$ tar czf version-log.tar.gz -C /var/log osgeolive
  • After the completion of the above script the new iso file is located in ~/livecdtmp along with the build logs. In case you wish to rerun the build process, do not remove or move the lubuntu official iso located in this folder to skip downloading it again.
  • It is required to reboot your host machine after build is completed
  • Once the ISO is complete copy it out to a server (a local server is fastest)
scp ~/livecdtmp/osgeolive-mini-8.0.iso user@server.org:destination/path/
scp ~/livecdtmp/osgeolive-mini-8.0-log.tar.gz user@server.org:destination/path/
  • And/OR wget or scp the file to the upload.osgeo.org server (Note wget is much faster if you have a good webserver to host from)

Build Full ISO from Mini ISO

Any mini iso file resulting from the procedure below can be turned into full iso, which is identical but adds the Windows and Mac software installers. Note that the resulting file may be bigger than a DVD. In that case the list of included applications in the load_mac_installers.sh need to be edited to exclude additional applications.

The standard current rules used to pick applications:

  • Desktop Applications
  • OSGeo projects and Incubated projects
sudo ./build_full_iso.sh /full/path/to/osgeo-live-mini-8.0.iso 2>&1 | tee ~/build_full_iso.log

How to do development / debugging with the current build method

We have created a debug build process so that projects can now easily create their own iso, including parts of the OSGeoLive (eg only one project) in order to test if the installer scripts work well under this new build method. Here are the steps to debug/test your application:

One time steps

You will need to create a pure Lubuntu Virtual Machine setup:

  • Download lubuntu-18.04-desktop-i386.iso from lubuntu web site.
  • Download and install VirtualBox.
  • Create a fresh VM installation of Lubuntu. You will need to create a virtual disk drive with at least 25 GB of space and allocate 768MB of RAM to the VM. During installation set the username to "user" and hostname to "osgeolive". DO NOT install system updates during lubuntu installation or after the installation is done. At this moment we work with the default kernel included in lubuntu.
  • After the VM is done, login as "user" and open a terminal.
  • Bootstrap the VM:
osgeolive$ cd /tmp
osgeolive$ wget https://github.com/OSGeo/OSGeoLive/raw/master/bin/bootstrap.sh
osgeolive$ chmod a+x bootstrap.sh
osgeolive$ sudo ./bootstrap.sh

This will install Git, the install scripts, and create a link to them from your home directory.

Steps to create the build (repeat as much as needed)

  • Make changes to your project's installation script and commit to Git.
  • Update the git code:
osgeolive$ cd ~/gisvm
osgeolive$ git pull origin master
  • Open file inchroot.sh with an editor and comment out all scripts you do not need for your test.
osgeolive$ cd ~/gisvm/bin
osgeolive$ vi inchroot.sh
  • Always leave un-commented the following scripts: setup.sh, install_services.sh, install_mysql.sh, install_java.sh, install_apache2.sh, install_tomcat6.sh, install_desktop.sh and setdown.sh.
  • Save your changes and execute the build:
osgeolive$ cd ~/gisvm/bin
osgeolive$ sudo ./build_chroot.sh i386 2>&1 | tee /var/log/osgeolive/chroot-build.log
  • After a while the iso will be created in ~livecdtmp/
  • Do not delete the file ~/livecdtmp/lubuntu-18.04-desktop-i386.iso as it will be needed for next build (saves time not to download again)
  • Logs are created at /var/log/osgeolive/chroot-build.log
  • Copy the iso and test

Build the Live DVD VM image

Create the VM

The OSGeoLive Virtual Machine creation process is now exactly similar to a plain Lubuntu VM installation. Use the mini iso file that was created from the previous chapter. Instructions can be found Live_GIS_Virtual_Machine

Package the VM

From within the VM, fill empty space with zeros in order to be able to shrink the virtual disk files: 

osgeolive$ sudo ~/gisvm/bin/zerofill.sh

Shrink the virtual machine: 

host$ VBoxManage modifyhd osgeolive.vdi --compact

Convert to vmdk format (more widely compatible): 

host$ VBoxManage clonehd osgeolive.vdi osgeolive-14.0-amd64.vmdk --format VMDK
OR with a recent version of QEMU
host$ qemu-img convert -f vdi -o compat6 -O vmdk osgeolive.vdi osgeolive-14.0-amd64.vmdk

Zip the image up: 

host$ 7z a -mx=9 osgeolive-14.0-amd64.vmdk.7z osgeolive-14.0-amd64.vmdk

Create the md5sum checksums, so which can be used to confirm that the images have been downloaded correctly: 

host$ md5sum *.7z*

Upload the Release

Upload to sourceforge

As of 6.0 the official releases are hosted on sourceforge. To upload you need a sourceforge account and permissions to the osgeo-live project upload. 

rsync -e ssh osgeolive-14.0-amd64.iso username,osgeo-live@frs.sourceforge.net:/home/pfs/project/o/os/osgeo-live/14.0/

Upload to the OSGeo Server

host$ scp -pr osgeolive-gisvm-2.0-alpha5 username@upload.osgeo.org:/osgeo/download/livedvd/

Update the index.html file at: https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk/download/index.html

Check the result at: http://download.osgeo.org/livedvd

Creating a torrent file

Ibiblio has offered torrent hosting, this section needs to include who to contact/how to setup Several tools exist for creating torrents. Below are the key settings you need.

Deluge with the torrent creator plugin has been tested.

  • Make sure you have a copy of the file you want to create a torrent for. Mounted remote drives may work.
  • Web Seed - URL to a web server that hosts the file, adds speed to the torrents especially at the start. (Most clients implement this now)
  • Tracker - There are 2 open and free trackers that have been tested, use one or the other as most clients do not handle multiple trackers yet.
http://tracker.openbittorrent.com/announce
udp://tracker.openbittorrent.com:80/announce

OR 

http://tracker.publicbt.com:80/announce
udp://tracker.publicbt.com:80/announce
  • Upload the .torrent file of your iso file for others to grab
  • Start the torrent on your machine or dedicated seeder, so that others have somewhere to start.
  • There is a possibility of seeding from osgeo machines but a Quality of Service (QoS) system would need to be in place to ensure it does not impact other services.

SVN branches and tags

Naming

  • main development happens in "trunk" (aka HEAD)
  • a branch is split off before release time. Bug fixes (only) happen in it and updates like version 2.1 are tagged from it.
its name might be like "arramagong_2"
  • a tag is a snapshot of the svn at some point in time, aka a release name. It does not make sense to checkin fixes to one. They should ideally be set as read-only after creation to prevent this.
its name might be like "release_20090927_arramagong_2_0"

Creating

cd livedvd/gisvm/

svn copy trunk branches/arramagong_2
svn commit branches/arramagong_2 -m "Splitting off branch for 2.x"

cd branches/arramagong_2/bin/
# something like this, maybe need a for loop or xargs:
sed -e 's+gisvm/trunk+gisvm/branches/arramagong_2+g' *.sh | less   # check
sed -i -e 's+gisvm/trunk+gisvm/branches/arramagong_2+g' *.sh       # execute
cd -
 
svn copy branches/arramagong_2 tags/release_20090927_arramagong_2_0_3
svn commit tags/release_20090927_arramagong_2_0_3 -m "tag release 2.0.3"

Merging

To merge a change from trunk into a release branch use "svn merge", as follows: (in this example r2131 from trunk) 

svn up branches/arramagong_2/
cd branches/arramagong_2/

svn merge -c 2131 https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk
svn diff
svn commit -m "bugfix: bikeshed should be orange! (merge from trunk r2131)"

If you do this a lot you might make a little shell script. Here's one called svn_merge_livefromtrunk.sh: 

#!/bin/sh
# Usage: svn_merge_livefromtrunk <rev number>
svn merge -c $1 https://svn.osgeo.org/osgeo/livedvd/gisvm/trunk

See also


Staging the new release

Website

Log in to live.osgeo.org. 

cd /osgeo/osgeolive
mkdir <versionnumber>

From another machine (ususally adhoc) go to the root of the docs and do 

scp -r /osgeo/livedvd/www/docs/* live.osgeo.org:/osgeo/osgeolive/<versionumber>/

Once done check that it's all good: 

http://live.osgeo.org/archive/versionnumber

Assuming docs are ok, then update the current symlink: 

chmod -R g+w <versionnumber>
rm current
ln -s <versionnumber> current

Download mirror

Upload files to https://sourceforge.net/projects/osgeo-live/files/

Add a new folder for the release, go into it, then scp the file to SourceForge.

Click the box to stage the files for 3 days to give time for the mirrors to populate.

Once the files are ready (no longer say "Pending...") and the download links and redirects from download.osgeo.org are tested & working, set the default download on the SourceForge site to either the full or mini ISO by clicking the "i" information button next to the file and "select all" for all platform types, then click the Save button.

Retrieved from "http://wiki.osgeo.org/w/index.php?title=Live_GIS_Build&oldid=126564"