RemoteAgent: Bridging Vague Human Intents and Earth Observation with RL-based Agentic MLLMs

TGRS 2026

Liang Yao1,*, Shengxiang Xu2,*, Fan Liu1,†, Chuanyi Zhang1, Bishun Yao1, Rui Min1, Yongjun Li1, Chaoqian Ouyang3, Shimin Di2, Min-Ling Zhang2
1Hohai University 2Southeast University 3Sun Yat-sen University

* Equal contribution. † Corresponding author.

Paper Code

Abstract

Earth Observation (EO) systems are essentially designed to support domain experts who often express their requirements through vague natural language rather than precise, machine-friendly instructions. Depending on the speciffc application scenario, these vague queries can demand vastly different levels of visual precision. Consequently, a practical EO AI system must bridge the gap between ambiguous human queries and the appropriate multi-granularity visual analysis tasks, ranging from holistic image interpretation to ffne-grained pixel-wise predictions. While Multi-modal Large Language Models (MLLMs) demonstrate strong semantic understanding, their text-based output format is inherently ill-suited for dense, precision-critical spatial predictions. Existing agentic frameworks address this limitation by delegating tasks to external tools, but indiscriminate tool invocation is computationally inefffcient and underutilizes the MLLM’s native capabilities. To this end, we propose RemoteAgent, an agentic framework that strategically respects the intrinsic capability boundaries of MLLMs. To empower this framework to understand real user intents, we construct VagueEO, a human-centric instruction dataset pairing EO tasks with simulated vague natural-language queries. By leveraging VagueEO for reinforcement ffne-tuning, we align an MLLM into a robust cognitive core that directly resolves image- and sparse region-level tasks. Consequently, RemoteAgent processes suitable tasks internally while intelligently orchestrating specialized tools via the Model Context Protocol exclusively for dense predictions. Extensive experiments demonstrate that RemoteAgent achieves robust intent recognition capabilities while delivering highly competitive performance across diverse EO tasks.

Architecture

RemoteAgent training and inference framework.
Overview of RemoteAgent. During training, the model is aligned via GRPO, guided by a uniffed multi-task reward that evaluates coordinate, numerical, and textual outputs. During inference, the agent dynamically plans and routes queries, directly resolving macroscopic tasks internally while delegating dense predictions to a specialized external toolkit. Task abbreviations: Visual Question Answering (VQA), Visual Grounding (VG), Classiffcation (CLS), Detection (DET), Segmentation (SEG), Referring Expression Segmentation (RES), Change Detection (CD), and Contour Extraction (CE).

🏆Contributions

  • Human-centric EO benchmark. VagueEO connects free-form user intents with standardized Earth Observation task annotations.
  • Capability-aware agent. RemoteAgent uses RL alignment to resolve intrinsic tasks internally while routing dense predictions to specialized tools.
  • Unified rewards. Coordinate, numerical, and textual outputs are supervised through a single GRPO-compatible reward interface.
  • Holistic evaluation. Experiments show strong intent recognition, data efficiency on intrinsic tasks, and expert-level precision on extrinsic tool calls.

Dataset

The paper constructs VagueEO, a benchmark that pairs vague, human-centric queries with standard structural annotations. Its training split covers intrinsic tasks used for RL fine-tuning, while its test split evaluates both intrinsic tasks and unseen dense spatial tasks.

10 EO Tasks

Scene classification, multi-label classification, grounding, counting, reasoning, detection, segmentation, referring segmentation, and change detection.

Vague Queries

LLM-generated free-form instructions simulate how non-expert users describe real analytical needs.

Structured Labels

Each query is paired with deterministic annotations across image-level labels, boxes, and masks.

VagueEO benchmark overview.
VagueEO Benchmark Overview. We construct ten diverse Earth Observation tasks that pair vague, human-centric queries with standardized structural annotations.

Main Results

RemoteAgent is evaluated along three axes: intent recognition, intrinsic MLLM capability, and extrinsic tool execution. It achieves strong sparse reasoning performance and routes dense tasks to tools with competitive precision.

Qualitative Analysis

RemoteAgent interprets free-form queries, identifies whether they require dense outputs, and invokes the relevant expert tool without manual intervention.

RemoteAgent qualitative routing examples.
Qualitative results of RemoteAgent. The agent accurately interprets free-form queries and dynamically routes them to specialized tools, seamlessly bridging vague intents with precision-critical execution.

BibTeX

@misc{yao2026RemoteAgent,
      title={RemoteAgent: Bridging Vague Human Intents and Earth Observation with RL-based Agentic MLLMs}, 
      author={Liang Yao and Shengxiang Xu and Fan Liu and Chuanyi Zhang and Bishun Yao and Rui Min and Yongjun Li and Chaoqian Ouyang and Shimin Di and Min-Ling Zhang},
      year={2026},
      eprint={2604.07765},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2604.07765}
}