PhilEO BENCH: EVALUATING GEO-SPATIAL FOUNDATION MODELS

[brunosan]

Might be our Benchmark framework

https://arxiv.org/pdf/2401.04464.pdf

Massive amounts of unlabelled data are captured by Earth Observation (EO) satellites, with the Sentinel-2 constellation
generating 1.6TB of data daily. This makes Remote Sensing a data-rich domain well suited to Machine Learning (ML) solutions. However, a bottleneck in applying ML models to EO is the lack of annotated data as annotation is a labour-intensive
and costly process. As a result, research in this domain has focused on Self-Supervised Learning and Foundation Model
approaches. This paper addresses the need to evaluate different Foundation Models on a fair and uniform benchmark by introducing the PhilEO Bench, a novel evaluation framework for EO Foundation Models. The framework comprises of a testbed and a novel 400GB Sentinel-2 dataset containing labels for three downstream tasks, building density estimation, road segmentation, and land cover classification. We present experiments using our framework evaluating different Foundation Models, including Prithvi and SatMAE, at multiple nshots and convergence rates.

[brunosan]

Key takeaways:

• This paper present a dataset "PhilEO Bench" of 400GB Sentinel-2 dataset containing labels for three downstream tasks, building density estimation, road segmentation, and land cover classification.
• Current state-of-the-art Earth Observation foundation models like SatMAE and Prithvi actually underperform to a simple Unet with skip connections at important downstream image-to-image tasks like building detection and land cover segmentation.
• This issue seems to come from the embeddings not being able to capture all needed fine semantics that a purpose-trained Unet would learn (but that purpose-trained Unet would not capture other semantics).

Implications for Clay:

I'm hopeful that #242 using a frozen encoder from a trained Clay can overcome these issues.

Other than that, I'm eager to test our model with this dataset and labels.