View source for TorchGeo DOFA FLORO



== DOFA Theory and Architecture Analysis ==

=== Core Design Principles ===
* Neuroplasticity-inspired: Based on brain's dynamic reorganization capacity in response to novel stimuli
* Wavelength-conditioned dynamic hypernetwork: Uses wavelength as unifying parameter across EO modalities
* Unified Transformer framework: Single architecture that handles diverse spectral bands and sensor modalities

=== Key Technical Components ===
1. '''Dynamic Hypernetwork''': Generates network weights based on central wavelengths of each spectral band
2. '''Shared Vision Backbone''': Universal feature learning module for all heterogeneous data modalities
3. Wavelength-aware Masked Image Modeling ('''MIM'''): Pretraining strategy that interpolates in weight space according to wavelength configurations

=== Key Classes: ===
1. <code>MaskedAutoencoderViT</code> - Main encoder class
2. <code>Dynamic_MLP_OFA</code> - Dynamic MLP layer for channel adaptation
3. <code>TransformerWeightGenerator</code> - For neuroplasticity-inspired weight generation

=== Architectural Features: ===
* Single unified ViT: Uses standard Vision Transformer backbone with modifications
* Dynamic MLP layers: <code>Dynamic_MLP_OFA</code> that adapts based on input channels
* Wavelength-aware processing: Uses <code>wave_lists</code> for different spectral band handling
* Neuroplasticity-inspired: Weight generation through transformer-based mechanism
* Channel-flexible design: Works with 2-202+ channels through dynamic layer adaptation

=== DOFA+ Enhancement ===
* Hierarchical Distillation Strategy: Preserves semantic priors from source model while guiding EO-specific pattern learning
* Dual Training Strategy: 
** Wavelength-aware MIM for EO-specific spatial patterns
** Hierarchical feature distillation for refining inherited semantic representations

=== MLP Layers ===
Looking at the DOFA code structure, ''dynamic MLP layers'' refers to a specific architectural component that adapts its parameters based on input characteristics:

==== Dynamic MLP Layers in DOFA: ====

* <code>Dynamic_MLP_OFA</code> - A specialized MLP (Multi-Layer Perceptron) layer that dynamically adjusts its weights and structure
* Unlike standard fixed MLPs, these layers can modify their internal parameters based on input features

How MLP Layers work:
1. Channel-adaptive processing': The MLP adapts to different input channel counts (2-202+ channels)
2. Wavelength-conditioned': Uses wavelength information to determine the appropriate weight 
configuration
3. Dynamic weight generation: Instead of fixed weights, the layer generates weights based 
on input characteristics

==== Implementation approach ====
*<code> TransformerWeightGenerator</code>: A component that dynamically generates network weights based on central wavelengths
* '''Hypernetwork''' concept: The dynamic MLP layer acts as a hypernetwork that produces weights for other layers
* Spectral band awareness: The layer structure changes to accommodate different spectral configurations

==== Purpose ====
The dynamic MLP layers allow DOFA to handle varying sensor specifications without requiring multiple fixed architectures. When input data has 2 channels (SAR), 3 channels (RGB), or 202 channels (hyperspectral), the same model architecture can adapt through these dynamic layers rather than needing separate models for each modality.


== FLORO Theory and Architecture Analysis ==

=== Core Design Principles ===
* Diversity-driven representation learning: Prioritizes sensor heterogeneity over corpus scale to enable cross-modal transferability
* Frozen-encoder evaluation protocol: Pretrained encoder remains fixed during downstream task adaptation, testing pure representational quality
* Masked autoencoding with modality-specific reconstruction: Learns latent structure by predicting masked tokens per input stream using lightweight decoders

=== Key Technical Components ===
1. '''FLOROGeoEncoder''': Main Vision Transformer encoder class for latent representation learning on grouped multimodal inputs
2. '''FLOROSSDecoder''': Masked autoencoding decoder used only during pretraining phase; reconstructs missing patches from visible context
3. '''FLOROLinearClassDecoder''': Task-specific linear classifier head trained under frozen-encoder protocol for downstream tasks

=== Key Classes: ===
1. <code>FLOROGeoEncoder</code> - Main Vision Transformer encoder class for latent representation learning
2. <code>FLOROSSDecoder</code> - Masked autoencoding decoder used only during pretraining phase; reconstructs missing patches from visible context
3. <code>FLOROLinearClassDecoder</code> - Task-specific linear classifier head trained under frozen-encoder protocol for downstream tasks

=== Architectural Features: ===
* Single unified ViT encoder: Processes grouped multimodal inputs without modality-specific branches or fusion layers; uses standard self-attention blocks with DropPath and LayerScale regularization
* Grouped channel structure: Optical and auxiliary streams are concatenated into fixed-width input tensors with validity masks indicating presence/absence of bands/modalities; processed via PatchEmbed layer that handles variable spectral groupings
* Masked token prediction: During pretraining, random patch masking occurs independently per token regardless of its originating modality; model reconstructs them using visible context from same or other modalities through FLOROSSDecoder
* Frozen-encoder transfer paradigm: Downstream tasks train only task-specific decoders (e.g., FLOROLinearClassDecoder) atop fixed encoder weights — no fine-tuning of base representation

=== FLORO+ Enhancement ===
* Curated multimodal diversity strategy: Uses ~80K samples spanning Sentinel-2, Sentinel-1, SkySat, UAV RGB/multispectral, and terrain products to maximize sensing condition variation rather than sample count
* Availability-aware processing: Validity channels explicitly signal which spectral groups or modalities are present per patch; implemented via NormalizeWithValidity transform that scales data using validity masks, enabling robust handling of incomplete sensor configurations during both pretraining and transfer

=== Input Representation ===
FLORO structures input as two primary streams with internal grouping:

==== Optical Stream ====
Contains reflectance values organized by spectral group plus validity indicators:
* Blue/Green/Red — visible spectrum bands
* Red Edge — vegetation stress indicator band
* NIR / NIR A — near-infrared for biomass and health assessment
* SWIR 1/SWIR 2 — shortwave infrared for moisture content analysis
* Validity Channels — binary flags per group indicating whether data is present or missing; processed via NormalizeWithValidity transform

==== Auxiliary Stream ====
Includes non-optical geospatial features:
* Elevation (DSM/DTM/DEM) — topographic structure information
* SAR VV/VH — dual-polarization radar backscatter for surface texture and moisture detection
* Validity Channels — same mechanism as optical stream to denote availability; processed via NormalizeWithValidity transform

==== Implementation approach ====
* Token grouping strategy: Each spectral or auxiliary group forms a contiguous block within the input tensor; positional embeddings are applied uniformly across all groups after PatchEmbed transformation
* Masking scheme: Random patch masking occurs independently per token regardless of its originating modality, forcing cross-modal contextual inference during FLOROSSDecoder reconstruction
* Decoder design: Separate lightweight MLP-based decoders map latent representations back to pixel space for each original input group — these are removed after pretraining; downstream tasks use linear heads like FLOROLinearClassDecoder

==== Purpose ====
The grouped channel architecture allows FLORO to ingest variable sensor combinations without architectural modification. When trained on full Sentinel-2 (13 bands) + SAR + elevation, it can later transfer to datasets containing only RGB or UAV-only imagery by leveraging learned cross-modal priors encoded in the shared latent space — enabled explicitly through validity masking during both pretraining and inference phase via NormalizeWithValidity transform.