Difference between revisions of "Osgeolive-jupyter"

From OSGeo
Jump to navigation Jump to search
Line 100: Line 100:
 
* rasterio :  not easy to build from source, crucial for modern sat imagery handling; sophisticated library design
 
* rasterio :  not easy to build from source, crucial for modern sat imagery handling; sophisticated library design
  
−
* Shapely/Fiona : direct link to libGEOS from python, example Notebooks in place   
+
* Shapely/Fiona : direct link to libGEOS/ogr from python; example Notebooks in place   
  
 
* mapclassify, lightgbm :  tricky to install, might have to wait until the versions are sorted here
 
* mapclassify, lightgbm :  tricky to install, might have to wait until the versions are sorted here

Revision as of 12:17, 20 February 2022

Focal Notes

Use PPA ubuntugis-unstable and default Ubuntu Universe dot-deb packages, the installation results look roughly like this: 

dbb@focalk:/etc$ cat lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS"
#--
 PGSQL="120" GEOS="3.10.1-CAPI-1.16.0" PROJ="8.2.0" LIBXML="2.9.10" LIBJSON="0.13.1"
#--

                                    Sat 19 Feb 2022 07:18:48 PM PST
$ jupyter --version
jupyter core     : 4.6.3
jupyter-notebook : 6.0.3
qtconsole        : not installed
ipython          : 7.13.0
ipykernel        : 5.2.0
jupyter client   : 6.1.2
jupyter lab      : not installed
nbconvert        : 5.6.1
ipywidgets       : 6.0.0
nbformat         : 5.0.4
traitlets        : 4.3.3

##-------------------------------
Python version: 3.8.10 (default, Nov 26 2021, 20:14:08) [GCC 9.3.0]
pandas version: 0.25.3
matplotlib version: 3.1.2
NumPy version: 1.17.4
SciPy version: 1.3.3
IPython version: 7.13.0
scikit-learn version: 1.0.2
##-----------------------------------------
dbb@focalk:~$ ipython3
Python 3.8.10 (default, Nov 26 2021, 20:14:08) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.13.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import sklearn                                                                                               
In [2]: from sklearn import show_versions                                                                            
In [3]: show_versions()                                                                                              

System:
    python: 3.8.10 (default, Nov 26 2021, 20:14:08)  [GCC 9.3.0]
executable: /usr/bin/python3
   machine: Linux-5.13.0-30-generic-x86_64-with-glibc2.29

Python dependencies:
          pip: 20.0.2
   setuptools: 45.2.0
      sklearn: 1.0.2
        numpy: 1.17.4
        scipy: 1.3.3
       Cython: 0.29.14
       pandas: 0.25.3      <== a weak spot in this setup
   matplotlib: 3.1.2
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
--
dbb@focalk:~$ jupyter --paths
config:
    /home/dbb/.jupyter
    /usr/local/etc/jupyter
    /etc/jupyter
data:
    /home/dbb/.local/share/jupyter
    /usr/local/share/jupyter
    /usr/share/jupyter
runtime:
    /home/dbb/.local/share/jupyter/runtime
--

Focal Jupyter Overview

Ipsum Lorem etc...

Install Strategy

the system OS python gets the major, stable parts first.. as a set .. ; next 

pip install --user  --upgrade-strategy=only-if-needed  py3-pkg

installs to the current user's local area, easy .. then I define a directory in `~/.jupyter/ notebooks` and start to put notebooks and their assets into that . It is certainly true that this install will be updated as more dot-debs are crafted, later this year.

Some priority python for the Jupyter Notebooks of 2022

  • GeoPandas (and pySAL) : direct link to GEOS 3.10, popular extension to Pandas data science; please recall that pySAL is merging into the GeoPandas project overall; GeoPandas+pySAL use matplotlib graphics base, prefers Folium for web maps (?)
  • Numpy and Pandas : fundamental base packaging, versions matter
  • PostGIS writing done with pyscopg2 (low-level) or geoalchemy2 (ORM based on sqlalchemy)
  • Scikit-Learn : original machine-learning library by international research team led by French math; stands the tests of time; integrated with SciPy for Fortran, BLAS, LAPACK performant libs; standard numpy for data allocation; important primitives like PCA, SVM, and KNN are optimized in practical ways, and then benchmarked against pure Fortran or C++ for comparison
  • boto for modern S3 and other data store, transfer; can use AWS credentials (or Hetzner, or..) modern version using pip installed locally
  • rasterio : not easy to build from source, crucial for modern sat imagery handling; sophisticated library design
  • Shapely/Fiona : direct link to libGEOS/ogr from python; example Notebooks in place
  • mapclassify, lightgbm : tricky to install, might have to wait until the versions are sorted here
  • LiDAR "no" : this is slightly painful since modern commercial GIS is flooded with this tech, delivered by advanced Western companies seeking something novel to sell. Little to no "libre" support and that includes here.
  • DeepLearning "no" : deep learning has taken over in the modern AI/ML research, for funding and hype, but not on this disk. Please note that the ML libraries on osgeolive are completely useful, accurate and performant today.. no need for a constant and fast Internet connection, nor a compute infrastructure the size of a large naval ship, or special impenetrable models that can only be built by those machines. -LINK- Serious and in-demand applications like "facial recognition" are not included here. Many real applications to remote sensing are part of DeepLearning, but are not available on this disk. Some benefits of not including DeepLearning include: no dependancies on tricky GPU libs; not suffering from fast-updates of important ML core libraries; avoid patent problems in Western Nations.

"I feel that a common and well-known distro like this is an opportunity to build engineering cooperation, between teams that may not interact otherwise.. like the physical code sprints" -darkblueb in California

Focal Setup Steps

  591  sudo apt install python3-dev python3-pip

  405  sudo apt install ipython3 python3-pandas python3-scipy
  408  sudo apt install r-cran-sf
  429  sudo apt install python3-shapely python3-fiona
  614  sudo apt install --yes python3-geoalchemy2

  670  sudo apt install --yes jupyter-core jupyter-client jupyter-console jupyter-nbformat jupyter-nbconvert jupyter-notebook python3-ipykernel
  673  sudo apt install python-notebook-doc jupyter


there is a trick .. pip install --user --upgrade-strategy=only-if-needed  py3pkg

  605  pip install --user wheel
  606  pip install --user lightgbm
  616  pip install --user eo-learn
  630  pip install --user folium

all [ rasterio, networkx, sklearn, boto, more  ] into    ~/.local/lib/python-3.8/

$ ./do_jupyter.sh
#---------------------------
#!/bin/bash

jupyter notebook --no-browser --ip=192.168.1.131 --notebook-dir=.jupyter/notebooks


~/.bashrc

##-
PATH=$PATH:~/.local/bin

Focal APT Repos

dbb@focalk:/etc/apt$ cat sources.list
# Automatically generated by Calamares on 2022-01-02.
# Lines starting with "deb" are mandatory, while lines starting with "deb-src"
# are for more detailed package information.

## See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
## newer versions of Lubuntu.
deb http://ubuntu.osuosl.org/ubuntu/ focal main restricted
deb-src http://ubuntu.osuosl.org/ubuntu/ focal main restricted

## Major bug fix updates produced after the final release of Lubuntu.
## Have you noticed a regression? Please report it!
## https://wiki.ubuntu.com/StableReleaseUpdates#Regressions

## Software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu team.
## Also, please note that software in Universe WILL NOT receive any review or
## updates from the Ubuntu security team directly. Updates in this repository
## are provided by volunteers, but most come from Debian.
deb http://ubuntu.osuosl.org/ubuntu/ focal universe
deb-src http://ubuntu.osuosl.org/ubuntu/ focal universe

## Software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu team,
## and may not be under a free licence. Please satisfy yourself as your rights
## to use the software. Also, please note that software in Multiverse WILL NOT
## receive any review or updates from the Ubuntu security team directly.
# deb http://us.archive.ubuntu.com/ubuntu/ focal multiverse
# deb-src http://us.archive.ubuntu.com/ubuntu/ focal multiverse
# deb http://us.archive.ubuntu.com/ubuntu/ focal-updates multiverse
# deb-src http://us.archive.ubuntu.com/ubuntu/ focal-updates multiverse

## Software from this repository contains tested security updates from the
## Ubuntu security team.
deb http://security.ubuntu.com/ubuntu focal-security main restricted
deb-src http://security.ubuntu.com/ubuntu focal-security main restricted
deb http://security.ubuntu.com/ubuntu focal-security universe
deb-src http://security.ubuntu.com/ubuntu focal-security universe
deb http://security.ubuntu.com/ubuntu focal-security multiverse
# deb-src http://security.ubuntu.com/ubuntu focal-security multiverse

## Software from this repository may not have been tested as extensively as
## software contained in the main release, although it includes newer versions
## of some applications which may provide useful features. Also, please note
## that software in Backports WILL NOT receive any review or updates from the
## Ubuntu security team.
deb http://ubuntu.osuosl.org/ubuntu/ focal-backports main restricted universe multiverse
deb-src http://ubuntu.osuosl.org/ubuntu/ focal-backports main restricted universe multiverse

## Uncomment the following two lines to add software from Canonical's
## "partner" repository.
## This software is not part of Ubuntu, but is offered by Canonical and the
## respective vendors as a service to Ubuntu users.
# deb http://archive.canonical.com/ubuntu focal partner
# deb-src http://archive.canonical.com/ubuntu focal partner

-------------------------

dbb@focalk:/etc/apt$ cat sources.list.d/ubuntugis-ubuntu-ubuntugis-unstable-focal.list

deb http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu focal main
deb-src http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu focal main
Retrieved from "http://wiki.osgeo.org/w/index.php?title=Osgeolive-jupyter&oldid=127930"