Note: This work is currently under review. The full dataset will be released progressively.

MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios

MobilityBench is a scalable benchmark for evaluating route-planning agents in real-world mobility scenarios. It is built from large-scale, anonymized mobility queries from Amap, organized with a comprehensive task taxonomy, and provides structured ground truth (required tool calls + verifiable evidence). All tool calls are executed in a deterministic replay sandbox for reproducible, multi-dimensional evaluation.

Scale & Coverage: 100,000 episodes across 22 countries and 350+ cities (including metropolitan areas), with a long-tailed geographic distribution.

Scenario Distribution (11 intents)

36.6% Basic Information Retrieval
9.6% Route-Dependent Information Retrieval
42.5% Basic Route Planning
11.3% Preference-Constrained Route Planning

Data Format

Field	Description
`query`	User query text
`context`	Context information (JSON, e.g., current location, city)
`task_scenario`	Fine-grained task category
`intent_family`	Coarse-grained intent category for evaluation aggregation
`tool_list`	Expected tool calls (JSON array)
`route_ans`	Ground truth route answer (JSON)

Sample Data (5 Examples)

Query	Task Scenario	Intent Family
去大石桥不走高速Go to Dashiqiao without taking the highway.	Option-Constrained Route Planning	Preference-Constrained Route Planning
现在成都大道会堵车吗？看一下地图，会不会堵Is Chengdu Avenue congested now? Looking at the map, is it likely to be congested?	Traffic Info Query	Basic Route Planning
我在哪Where am I?	Geolocation Query	Basic Information Retrieval
知道离滇池会展中心有多远How far it is from Dianchi Convention and Exhibition Center?	Route Property Query	Route-Dependent Information Retrieval
到寨河收费站入口不走高速To reach the Zhaihe toll station entrance without taking the highway.	Option-Constrained Route Planning	Preference-Constrained Route Planning

Citation

If you use this dataset in your research, please cite the following paper:

@article{song2026mobilitybench,
  title={MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios},
  author={Song, Zhiheng and Zhang, Jingshuai and Qin, Chuan and Wang, Chao and Chen, Chao and Xu, Longfei and Liu, Kaikui and Chu, Xiangxiang and Zhu, Hengshu},
  journal={arXiv preprint arXiv:2602.22638},
  year={2026}
}