TorchGeo embeddings

From OSGeo
Jump to navigation Jump to search

Template:Infobox Paper

[arXiv:2601.13134v1] [cs.SE] 19 Jan 2026

Earth Embeddings as Products: Taxonomy, Ecosystem, and Standardized Access is a comprehensive survey that organizes existing geospatial embedding products into a structured taxonomy through a three-layer taxonomy: Data, Tools, and Value. This research paper provides a detailed metadata atlas (resolution, license, etc.). It also proposes a unified integration by implementing standardized data loaders for these embeddings in [TorchGeo] .

An overview landscape is proposed comprising: a) Analysis Frameworks & Tools b) Embeddings data artifacts c) Charting downstream application value, specifically mapping tasks and retrieval tasks.

Embeddings are differentiated as either location-typed, patch-typed, or pixel-typed. Details of existing products are shown. "We extend TorchGeo with a unified API that standardizes the loading and querying of diverse embedding products."

1. Foundation Models for Earth Observation (EO)

These are the leading projects that aim to build general-purpose models capable of representing Earth from satellite imagery and other geospatial modalities.

Projects

  • OlmoEarth – [AllenAI] (2025)
    Latent image modeling approach for multimodal Earth observation.
  • Major TOM – [MajorTOM] AFrancis IGARSS 2024
    (Terrestrial Observation Metaset) is an open-source framework and massive satellite dataset created by the European Space Agency's Φ-lab.
  • Clay Foundation Model – [HuggingFace] (2024)
    A multimodal foundation model for Earth using diverse data sources.
  • Earth Index Embeddings – [EarthGenome] (2025)
    A large-scale embedding system built from Earth observation data.
  • Copernicus-Embed – [LINK] Zhu et al., AI4Copernicus Project
    Foundation model leveraging Copernicus Sentinel data.
  • Presto Embeddings – [NASAHarvest]
    Embedding framework for satellite time series and land use analysis.
  • Tessera Embeddings – [GeoTessera] Docs / [REPO]
    Pixel-based Temporal spectral embeddings for Earth representation.
  • Google Satellite Embedding (AlphaEarth) – [LINK] Google Earth Engine
    An early-stage embedding model using Google's global satellite data.

Key Papers

  • XXZhu 2025 [LINK] "On the Foundations of Earth Foundation Models" – Nature Computational Science
  • CFBrown 2025 [LINK] "AlphaEarth Foundations"
  • KKlemmer 2023 [LINK] "SatCLIP: Global Location Embeddings with Satellite Imagery"

2. Datasets

Large-scale, open-access datasets play a central role in training and evaluating Earth foundation models. See the list below for a representative sample.

3. Model Architectures for Embeddings

The generation of robust Earth observation embeddings relies on specific model families pre-trained on large-scale geospatial data. The following architectures are central to the TorchGeo ecosystem and recent foundation model research, categorized by their primary learning paradigm.

Vision-Language Models

These models align visual satellite imagery with textual or geographic metadata, creating unified embedding spaces for cross-modal retrieval.

  • SatCLIP – [[LINK]] (AAAI 2025)

A vision-language model designed to generate global location representations by aligning satellite imagery with geographic text embeddings, originally published in 2023 and widely studied.

A global dataset for multi-modal and geospatial representation learning. It consists of 12 modalities including pixel-level and image-level modalities from 1.2 million locations.

Self-Supervised Vision Transformers

Transformers pre-trained via self-supervised objectives (e.g., DINO, MAE) are currently the state-of-the-art for extracting rich, transferable embeddings from remote sensing imagery.

  • DINO / DINOv2 / DINOv3 – [[LINK]] (Meta/INRIA, 2021–2023)

Vision transformers trained with self-distillation without labels. DINOv2, in particular, produces highly discriminative embeddings that capture fine-grained geospatial features without downstream fine-tuning.

  • MAE (Masked Autoencoders) – [[LINK]] (CVPR 2021)

A self-supervised encoder that learns robust image representations by reconstructing masked patches. When applied to satellite data, MAE embeddings capture global context and local texture effectively.

Efficient Convolutional Backbones

While Transformers dominate embedding research, optimized CNNs remain critical for efficient embedding extraction in resource-constrained environments or as feature extractors for hybrid models.

  • ConvNeXt V2 – [[LINK]] (Woo et al., CVPR 2023)

A modernized ConvNet that incorporates Transformer-like design principles (e.g., large kernel sizes, layer normalization). It serves as a high-performance, efficient backbone for generating spatial embeddings in TorchGeo.

  • ResNet – [[LINK]] (He et al., CVPR 2016)

The foundational residual network. While older, ResNet variants remain the standard baseline for comparing embedding quality in early-stage Earth Foundation Model development.

Distillation & Advanced Approaches

  • Distillation methods – Transfer knowledge from large models.
  • Neural plasticity-inspired models – TorchGeo_DOFA: Inspired by biological learning mechanisms.
  • Multi-label guided soft contrastive learning – YWang, IEEE TGRS, 2024.
  • Barlow Twins – Method for learning representations without contrastive loss.
  • Continual Barlow Twins – Extends Barlow Twins to continual learning in EO segmentation.

4. Tools & Benchmarks

These are software systems and frameworks that support development, evaluation, or deployment of EO AI models.

Tools

PyTorch library for geospatial deep learning.

  • NeuCo-Bench – [LINK] RVinge, arXiv 2025

Benchmarking framework for neural embeddings in Earth observation.

  • GeoINRID – [LINK] GitHub: arjunarao619/GeoINRID

Geospatial inference and representation learning toolkit.

Challenges

  • Embed2Scale Challenge – [LINK] CVPR CAlbrecht 2025

Large-scale Earth vision challenge focused on scale-aware embeddings.

  • TerraMind Blue-Sky Challenge –

Generative modeling for Earth observation.

5. Key Themes & Trends

  1. Foundation Models: TorchGeo now includes data loaders designed for search/retrieval (Clay, Major TOM, Earth Index), and for dense prediction tasks like land cover mapping (Copernicus, Presto, Tessera, Google). TorchGeo allows us to enable fair, side-by-side benchmarking of different embedding models on the same downstream tasks, forming the basis for future experiments. Projects are encouraged to strengthen and improve explainability.
    • Major TOM Notes: Major TOM embeddings are not (yet) really product-oriented and are aimed with a similar purpose to the MT Core datasets - to make it easier to experiment and benchmark model outputs (hence, unlike TESSERA and AEF which came a few months after, MT embeddings do not have consistent or aggregated temporal scope). We haven't had enough time to finish off the preprint, but my current plan is to provide a simple MT Embedding benchmark at this year's EGU and integrate that into the arxiv pre-print. --Miko
    • Earth Index / Earth Genome: Use the Earth Index application (earthindex.ai) for non-technical users to use the embeddings we published on source.coop. Users of the web app (non-technical journalists, indigenous communities/allies, NGOs) have been our main focus. Users of the source.coop embeddings have generally been more technical folks interested in exploring/innovating in what's possible --BenStrong
    • Clay: Clay and Presto offer documented tutorials on generating new embeddings with their models. In CLAY, the encoder receives unmasked patches, latitude-longitude data, and timestep information. Notably, the last 2 embeddings from the encoder specifically represent the latitude-longitude and timestep embeddings.
  1. Self-Supervised Learning (SSL):
  1. Multimodal Integration:
  1. Open Data & Tools: Open-source projects (e.g., TorchGeo, Copernicus-Embed) and public datasets (EuroSAT, EuroCrops) are crucial for reproducibility and democratization of EO AI. Projects are encouraged to increase Input Data Diversity, and to adopt cloud-native data formats for geospatial data.
  1. Benchmarking: Projects are encouraged to standardize in benchmarking. Benchmarks including NeuCo-Bench and Embed2Scale.


Deatasets Auto-Edit

Land use classification dataset using Sentinel-2 satellite data.

Crop type mapping dataset for Europe.

  • National Land Cover Database (NLCD) – [MRLC]

USA land cover classes.

Multimodal, multitemporal dataset for self-supervised learning.

  • Copernicus-Pretrain – [GitHub]

An extension of the SSL4EO-S12 dataset to all major Sentinel missions (S1-S5P).

Large-scale multi-label satellite image classification dataset.

Remote sensing image classification dataset with 45 categories.

Aerial image dataset for land use classification.

Semantic segmentation dataset for urban areas from aerial imagery.

Semantic segmentation dataset for urban areas from aerial imagery.

  • Inria Aerial Image Labeling – [Inria]

Aerial image segmentation dataset for building footprint extraction.

National Agriculture Imagery Program data for the USA.

Multispectral imagery from the Sentinel-2 mission.

Long-term archive of medium-resolution satellite imagery.

Collaborative project to create a free editable map of the world.

Global dataset of biomass burning emissions.

Global biodiversity information facility dataset.

Global building footprint detection dataset.

Open-source aerial imagery dataset.

Large-scale remote sensing image classification dataset.

Pre-trained embeddings for Google satellite imagery.

Large-scale dataset for object detection in aerial images.


Research Directions

  • Unified Earth Foundation Models:
  • Interpretability in EO AI: Exploring how these embeddings can be interpreted by domain experts.
  • Ethics and Bias: Investigating fairness and bias in global EO models trained on unevenly distributed data.
  • Edge Deployment: Making these large foundation models deployable on resource-constrained platforms (e.g., for field use).
Retrieved from "http://wiki.osgeo.org/w/index.php?title=TorchGeo_embeddings&oldid=136098"