Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle

Jiayu Wang, Weijiang Lv, Bowen Fu, Jing Fu, Jiayi Song, Lingyu Zhang, Lanxuan Xue, Luodi Chen, Zepeng Xin, Kaiyu Li, Xiangyong Cao

• Launch the AARR benchmark series, a novel framework for evaluating the capabilities of LLM agents in authentic research scenarios.
• Propose AARRI-Bench, the inaugural benchmark in this series, which comprises tasks designed to simulate real research intern activities.
• Conduct extensive experiments across frontier models and agentic systems, providing a comprehensive analysis of their current capabilities and limitations.

Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation

Koutian Wu, Junjie Zhou, Ergan Shang, Jiayu Wang, Pengqian Han, Junkai Wang, Wanghan Xu, Lin Shi

• Combine daily discovery of benchmark papers, repositories, datasets, and releases with a searchable benchmark catalog, mentions in model cards and technical reports, and score histories.
• Daily discovery draws on 37 sources: 13 direct connectors and 24 first-party research and engineering feeds.
• Release the web dashboard with a benchmark leaderboard, a Pareto frontier view of score against measured use, saturation and trend views, daily feeds, downloadable evidence, a command-line interface (CLI) for offline queries, and reproducible analysis.

GeoFaith: A Spatio-Temporal Dual View of Faithful Chain-of-Thought

Paper

Weijiang Lv, Wentong Zhao, Jiayu Wang, Yuhao Wu, Jiaheng Wei, Xiaobo Xia

• Introduce a spatio-temporal perspective on Chain-of-Thought faithfulness, characterizing reasoning via latent geometric structure and entropy dynamics.
• Develop a scalable bootstrapping pipeline that expands step-level faithfulness annotations from 1k to 20k examples across multiple domains, enabling large-scale process supervision.
• Design a faithfulness-aware reinforcement learning framework that jointly optimizes outcome correctness, process faithfulness, and trajectory consistency, achieving superior detection and generation performance while producing shorter, more interpretable reasoning chains.

Designing Domain-Specific Agents via Hierarchical Task Abstraction Mechanism

Kaiyu Li, Jiayu Wang, Zhi Wang, Hui Qiao, Weizhan Zhang, Deyu Meng, Xiangyong Cao

• We introduce a novel agent design framework centered on a Hierarchical Task Abstraction Mechanism (HTAM).
• We instantiate this framework as EarthAgent, a multi-agent system tailored for complex geospatial analysis. To evaluate such complex planning capabilities, we build GeoPlan-bench, a comprehensive benchmark of realistic, multi-step geospatial planning tasks. It is accompanied by a suite of carefully designed metrics to evaluate tool selection, path similarity, and logical completeness.
• Experiments show that EarthAgent substantially outperforms a range of established single- and multi-agent systems.

SPD-Faith Bench: Diagnosing and Improving Faithfulness in Chain-of-Thought for Multimodal Large Language Models

Weijiang Lv，Yaoxuan Feng，Xiaobo Xia，Jiayu Wang，Yan Jing，Wenchao Chen，Bo Chen

• We introduce SPD-Faith Bench, a diagnostic benchmark based on fine-grained image difference reasoning that enforces explicit visual comparison.
• Evaluations on state-of-the-art MLLMs reveal two systematic failure modes, perceptual blindness and perception-reasoning dissociation.
• We propose SAGE, a train-free visual evidence-calibrated framework that improves visual routing and aligns reasoning with perception. Results highlight the importance of explicitly evaluating faithfulness beyond response correctness

DescribeEarth: Describe Anything for Remote Sensing Images

Kaiyu Li, Zixuan Jiang, Xiangyong Cao, Jiayu Wang, Yuchen Xiao, Deyu Meng, Zhi Wang

• We propose Geo-DLC, a novel task of object-level fine-grained image captioning for remote sensing.
• We construct DE-Dataset, a large-scale dataset containing 25 categories and 261,806 annotated instances with detailed descriptions of object attributes, relationships, and contexts.
• Furthermore, we introduce DE-Benchmark, an LLM-assisted question-answering based evaluation suite designed to systematically measure model capabilities on the Geo-DLC task.
• We also present DescribeEarth, a Multi-modal Large Language Model (MLLM) architecture explicitly designed for Geo-DLC.
• Our DescribeEarth model consistently outperforms state-of-the-art general MLLMs on DE-Benchmark, demonstrating superior factual accuracy, descriptive richness, and grammatical soundness, particularly in capturing intrinsic object features and surrounding environmental attributes across simple, complex, and even out-of-distribution remote sensing scenarios.