RemoteCLIP

RemoteCLIP: A Vision Language Foundation Model for Remote Sensing

TGRS 2024

Fan Liu^1,*†, Delong Chen^2,*†, Zhangqingyun Guan¹, Xiaocong Zhou¹, Jiale Zhu¹, Qiaolin Ye³, Liyong Fu⁴, Jun Zhou⁵

¹Hohai University ²HKUST ³Nanjing Forestry University ⁴Chinese Academy of Forestry ⁵Griffith University

* Equal contribution. † Corresponding authors.

Abstract

General-purpose foundation models have led to recent breakthroughs in artificial intelligence (AI). In remote sensing, self-supervised learning (SSL) and masked image modeling (MIM) have been adopted to build foundation models. However, these models primarily learn low-level features and require annotated data for fine-tuning. Moreover, they are inapplicable for retrieval and zero-shot applications due to the lack of language understanding. To address these limitations, we propose RemoteCLIP, the first vision-language foundation model for remote sensing that aims to learn robust visual features with rich semantics and aligned text embeddings for seamless downstream application. To address the scarcity of pretraining data, we leverage data scaling which converts heterogeneous annotations into a unified image-caption data format based on box-to-caption (B2C) and mask-to-box (M2B) conversion. By further incorporating unmanned aerial vehicle (UAV) imagery, we produce a 12× larger pretraining dataset than the combination of all available datasets. RemoteCLIP can be applied to a variety of downstream tasks, including zero-shot image classification, linear probing, k-NN classification, few-shot classification, image–text retrieval, and object counting in remote sensing images. Evaluation of 16 datasets, including a newly introduced RemoteCount benchmark to test the object counting ability, shows that RemoteCLIP consistently outperforms baseline foundation models across different model scales. Impressively, RemoteCLIP beats the state-of-the-art (SOTA) method by 9.14% mean recall on the RSITMD dataset and 8.92% on the RSICD dataset. For zero-shot classification, our RemoteCLIP outperforms the contrastive language image pretraining (CLIP) baseline by up to 6.39% average accuracy on 12 downstream datasets.

Architecture

RemoteCLIP data scaling, pretraining, and downstream evaluation pipeline.

Overview of the RemoteCLIP pipeline. Step 1: RemoteCLIP is trained on a diverse collection of remote sensing datasets, covering ten object detection datasets (DET-10, six of them are satellite imaginary datasets and four of them are UAV datasets), four remote sensing semantic segmentation datasets (SEG-4), and three remote sensing image–text datasets. We propose B2C generation and M2B conversion to fully utilize heterogeneous annotations, and scale up the training data to 12× of the combination of all involved image–text data. Step 2: We perform continual pretraining based on the CLIP model, specializing it in the remote sensing domain. Step 3: We perform a comprehensive evaluation on seven tasks using 16 downstream datasets, including a newly created RemoteCount dataset, to demonstrate the strong capability and generalization ability of RemoteCLIP.

🏆Contributions

Large-scale remote sensing image-text data. The paper constructs a pretraining corpus 12 times larger than the combination of RSITMD, RSICD, and UCM.
Vision-language foundation model. RemoteCLIP aligns visual and textual representations to learn semantic remote sensing features for zero-shot and retrieval use cases.
Diverse downstream evaluation. The model is evaluated on retrieval, zero-/few-/full-shot classification, k-NN classification, and object counting.
RemoteCount benchmark. The work introduces a counting benchmark to test fine-grained language understanding in remote sensing imagery.

Dataset

RemoteCLIP builds a large-scale pretraining dataset by unifying retrieval, detection, and segmentation data. Detection annotations are converted into captions through Box-to-Caption generation, while segmentation masks are first converted into boxes and then captions. The final training set contains 165,745 images and 828,725 image-text pairs.

RET-3

Human-captioned retrieval datasets: RSICD, RSITMD, and UCM.

DET-10 + SEG-4

Detection and segmentation annotations are converted into captions for scalable pretraining.

RemoteCount

A new 947-pair benchmark covering 13 categories for object counting from 1 to 10.

RemoteCLIP dataset and downstream benchmark distribution.

Training and evaluation data distribution after RemoteCLIP data scaling.

RemoteCount samples for evaluating object counting ability.

Main Results

RemoteCLIP improves cross-modal retrieval, zero-shot classification, linear probing, k-NN classification, and object counting. Its gains are consistent across ResNet and Vision Transformer backbones.

Zero-shot classification results on 12 datasets.

Zero-shot classification: RemoteCLIP improves average accuracy across 12 remote sensing datasets.

Linear probing and k-NN classification results on 12 downstream datasets.

Object counting: RemoteCLIP is more robust than CLIP on RemoteCount.

BibTeX

@article{liu2024remoteclip, title={Remoteclip: A vision language foundation model for remote sensing}, author={Liu, Fan and Chen, Delong and Guan, Zhangqingyun and Zhou, Xiaocong and Zhu, Jiale and Ye, Qiaolin and Fu, Liyong and Zhou, Jun}, journal={IEEE Transactions on Geoscience and Remote Sensing}, volume={62}, pages={1--16}, year={2024} }