SAC:Repo

From OSGeo
Jump to navigation Jump to search

OSGeo provides a https://repo.osgeo.org/ manager to support our projects in distribution of programming artifacts (see also Download Server ). The repository manager is a structured web service intended for consumption by build systems and can be configured to work with Maven, Docker, npm, and others.

Maven

How to use OSGeo Maven repository

To make use of repo.osgeo.org to access OSGeo java projects.

Example pom.xml file 

  <repositories>
    <repository>
      <id>osgeo-release</id>
      <name>OSGeo Repository</name>
      <url>https://repo.osgeo.org/repository/release/</url>
      <!-- Contains releases for OSGeo projects: GeoTools, GeoWebCache, GeoServer, ...     -->
      <!-- Includes third-party-dependences, and caches of:                                -->
      <!-- ucar (https://artifacts.unidata.ucar.edu/content/repositories/unidata-releases) -->
      <!-- geosolutions (http://maven.geo-solutions.it/)                                   -->
      <snapshots>
        <enabled>false</enabled>
      </snapshots>
      <releases>
        <enabled>true</enabled>
      </releases>
    </repository>
  </repositories>

How to fix a broken build

If your project previously used:

Your build will fail with an error similar to: 

[ERROR] Failed to execute goal on project gt-opengis: Could not resolve dependencies for project org.geotools:gt-opengis:jar:21-SNAPSHOT: The following artifacts could not be resolved: javax.media:jai_codec:jar:1.1.3, javax.media:jai_core:jar:1.1.3: Failure to find javax.media:jai_codec:jar:1.1.3 in http://download.osgeo.org/webdav/geotools/ was cached in the local repository, resolution will not be reattempted until the update interval of osgeo has elapsed or updates are forced -> [Help

Please update your build, replacing: 

<repository>
    <id>osgeo</id>
    <name>Open Source Geospatial Foundation Repository</name>
    <url>http://download.osgeo.org/webdav/geotools/</url>
</repository>

With the following: 

<repositories>
  <repository>
    <id>osgeo</id>
    <name>OSGeo Release Repository</name>
    <url>https://repo.osgeo.org/repository/release/</url>
    <snapshots><enabled>false</enabled></snapshots>
    <releases><enabled>true</enabled></releases>
  </repository>
</repositories>

Alternative: Update settings.xml to mirror webdav and boundlessgeo repositories: 

<mirrors>
  <mirror>
    <id>osgeo-release</id>
    <name>OSGeo Repository</name>
    <url>https://repo.osgeo.org/repository/release/</url>
    <mirrorOf>osgeo</mirrorOf>     <!-- previously http://download.osgeo.org/webdav/geotools/ -->
  </mirror>
  <mirror>
    <id>geoserver-releases</id>
    <name>Boundless Repository</name>
    <url>https://repo.osgeo.org/repository/Geoserver-releases/</url>
    <mirrorOf>boundless</mirrorOf> <!-- previously http://repo.boundlessgeo.com/main/ -->
  </mirror>
</mirrors>


For additional information, including how to make use of SNAPSHOTS, please see GeoTools User Guide Upgrade Instructions.

Maven Instructions for OSGeo Projects

OSGeo Projects (and community projects) are encouraged to make use of the repo.osgeo.org for releases:

0. Before you start, you will need an OSGeo User ID

1. Create a TRAC ticket https://trac.osgeo.org/osgeo requesting repo access with the name of your project, and OSGeo UserID requiring access. 

  A project repository (maphappy-releases) will be setup, with access granted to the OSGeo UserID requested.

2. Update your pom.xml repositories with osgeo release repository. 

  <repositories>
    <repository>
      <id>osgeo-release</id>
      <name>OSGeo Repository</name>
      <url>https://repo.osgeo.org/repository/release/</url>
      <!-- Contains releases for OSGeo projects: GeoTools, GeoWebCache, GeoServer, ...     -->
      <!-- Includes third-party-dependences, and caches of:                                 -->
      <!-- ucar (https://artifacts.unidata.ucar.edu/content/repositories/unidata-releases) -->
      <!-- geosolutions (http://maven.geo-solutions.it/)                                   -->
      <snapshots><enabled>false</enabled></snapshots>
      <releases><enabled>true</enabled></releases>
    </repository>
  </repositories>

3. Update your pom.xml distribution management with project releases repository. 

  <distributionManagement>
    <repository>
      <id>maphappy</id>
      <name>OSGeo GeoServer Release Repository</name>
      <url>https://repo.osgeo.org/repository/maphappy-releases/</url>
    </repository>
  </distributionManagement>

4. Update your settings.xml with OSGeo UserID credentials 

    <server>
      <id>osgeo</id>
      <username>maphappy</username>
      <password>Foss4gForever</password>
    </server>

5. Advanced: Your project releases repository should be usable to synchronize your artifacts to maven central. We are looking at becoming an approved forge for maven central upload.

7. Advanced: For projects running a build system (jenkins, GitHub actions) you may wish set up a snapshot repository for your team. 

    <repository>
      <id>osgeo-snapshot</id>
      <name>OSGeo Repository</name>
      <url>https://repo.osgeo.org/repository/snapshot/</url>
      <snapshots><enabled>true</enabled></snapshots>
      <releases><enabled>false</enabled></releases>
    </repository>

  <distributionManagement>
    <snapshotRepository>
      <id>maphappy</id>
      <name>OSGeo GeoServer Snapshot Repository</name>
      <url>https://repo.osgeo.org/repository/maphappy-snapshots/</url>
     </snapshotRepository>
  </distributionManagement>

If you have any questions please use your TRACK ticket for communication.

For a working example see pom.xml from the GeoWebCache project.

Docker

OSGeo provides a nexus based docker registry for Projects to use.

Current docker repositories are as follows: Group Repositories

  • docker - docker.osgeo.org (this groups all the docker project repositories, and should be used just for docker pull requests)

Project hosted repositories (should be used only for pushes)

  • PostGIS - postgis-docker.osgeo.org (writes to repo postgis-docker) - PostGIS docker development guide decribing PostGIS project use.
  • Geos - geos-docker.osgeo.org (writes to repo geos-docker)
  • Geoserver - geoserver-docker.osgeo.org (writes to repo geoserver-docker)

Note for pulling, all pulling should be done via docker.osgeo.org and that allows anoymous pull. All the projects are grouped together under the docker group. Pushing needs to be done via the project specific docker domains by first logging into the respective domains docker login <project-docker-domain> with your OSGeo LDAP account or build user login. As required by docker, in order to push an image, the repo of the image must be prefixed with the project domain e.g. postgis-docker.osgeo.org/postgis/build-test:alpine would be the name we might tag a build so we can push it to postgis-docker.osgeo.org.

An image pushed to any project docker repo can be pulled via docker.osgeo.org. Example 

  docker pull docker.osgeo.org/postgis/build-test:alpine

Would pull the image pushed to postgis-docker.osgeo.org.


Docker Instructions for OSGeo Projects Teams

OSGeo Projects (and community projects) are encouraged to make use of the repo.osgeo.org for docker:

0. Before you start, you will need an OSGeo User ID

1. Create a TRAC ticket https://trac.osgeo.org/osgeo requesting repo access with the name of your project, and OSGeo UserID requiring access.


As an example see this ticket , leading to the creation of `postgis-docker` repository, with access granted to a `postgis-build` and `robe` OSGeo userids.

Nexus

The repo.osgeo.org is currently managed as an instance of Sonatype Nexus Repository Manager as shown below.

  • This is a repository manager intended for use by build systems
  • If you like you can use search to locate specific artifacts
  • Nexus also offers a chance to browse contents with an html view
  • Artifact URLs are stable and can be used as an alternative to downloads.osgeo.org and does provide download counts.

Sonatype Nexus Repository Manager

Repositories

Maven repository naming conventions:

  • project-releases (hosted) - each project manages its releases in a distinct hosted repository. They may wish to upload third party dependencies here also.
  • project-snapshots (hosted) - each project with build automation manages its snapshots in a distinct hosted repository, subject to a policy expiring artifacts after an appropriate time
  • external-cache (proxy) - external repositories can be cached at repo.osgeo.org to improve build times. Each external repository subject to routing policies to limit contents stored.
  • release (group) - Gathering all releases (from multiple project releases repositories, upstream cached repositories) providing a single repository for the community
  • snapshot (group) - Gather all snapshots from osgeo projects into a single repository for the community

Docker repository naming conventions:

  • project-docker (hosted) - each project manages images in a distinct hosted repository
  • docker (group) - Gather all docker projects into a single repository for the community to pull from

Nexus stores artifacts in blob storage (presently on the file system). We are defining different blob stores for different kinds of content:

  • releases - osgeo projects artifacts
  • docker - policy will be defined in conjunction with initial projects making use of this service
  • snapshots - publish transit artifacts produced from the active development branches (See automation below for cleanup policies and tasks)
  • cache - isolated into separate storage so we have the option of avoid backups in the future

Automation:

  • snapshot_policy (cleanup policy) - unused as we have switched to using tasks below. We experimented with defining cleanup policies to manage snapshots. Setting a limit of five days removed artifacts on maintenance branches that are updated infrequently; setting a "last downloaded before 15 days" results in too much storage used by active branches.
  • daily: maven delete snapshot (task) - set to clean up snapshots after five days, while maintaining a minimum of 1 snapshot
  • daily: publish maven indexer files (task) - used to publish an index used by build tools to synchronize thier understanding of the artifacts we publish
  • daily: delete incomplete docker uploads (task)
  • hourly: clean up repositories (task) - run hourly to apply cleanup policies to our repositories

Routing policies:

  • Routing policies have been defined to minimize the amount of artifacts cached from upstream repositories

Users and groups

We anticipate a very small number of users with access to this system, primarily projects steering committee members, and artificial users created for individual build processes.

  • Users are added via LDAP so your OSGeo User ID is required
  • Build processes: User is created locally on the Nexus instances with access to a project specific repository.
  • Groups, while LDAP groups are available the small number of users has not warranted making use of this facility
Retrieved from "http://wiki.osgeo.org/w/index.php?title=SAC:Repo&oldid=123908"