MobilePA-Bench is an interactive, stateful, and tool-centric benchmark for evaluating the tool-calling and planning capabilities of mobile planner agents. It moves beyond static function matching by executing agent actions in a mutable mobile environment and checking both the action trace and the resulting state.
- Executable and stateful: tool calls run against an environment whose application data, permissions, and device state evolve after every action.
- Broad mobile coverage: 1,705 evaluation tasks exercise 212 realistic tools across 13 functional domains and 89 subcategories.
- Four capability dimensions: Tool Use, Memory Usage, Skill Usage, and Sub-agent Collaboration.
- Evidence-based evaluation: fixed policies verify tool selection, grounded arguments, execution order, final environment state, and agent behavior.
- Realistic failure modes: agents must handle tool dependencies, permission boundaries, conflicting requests, runtime errors, and incomplete user context.
- 2026-08-25: The project repository was opened with an interactive project page, leaderboard, and a private-evaluation link.
- 2026-08-24: The MobilePA-Bench paper was released on arXiv.
MobilePA-Bench models a mobile planning agent as a decision-maker operating through structured tools, reusable skills, persistent memory, and specialized sub-agents. The environment executes each action, updates its state, and returns observations or runtime errors that the agent must incorporate into subsequent decisions.
The evaluator assigns a fixed verification policy to each task. Depending on the task, success can require an exact tool call, a target state transition, a prescribed action order, or a valid collaboration pattern. This makes the benchmark sensitive to whether an agent actually completes a request—not merely whether it produces a plausible-looking response.
| Dimension | What it measures |
|---|---|
| Tool Use | Grounded tool selection, argument construction, ordered execution, recovery, and safe refusal |
| Memory | Retrieval and application of user profiles, preferences, routines, history, and situational context |
| Skills | Selection and execution of reusable composite procedures instead of rebuilding every workflow from scratch |
| Sub-agent | Task decomposition, contextual handoff, and coordination with GUI, search, image, and other specialized agents |
We offer a confidential evaluation track for hosted mobile planner agents. Submit an HTTPS, OpenAI-compatible endpoint with tool-calling support, and MobilePA-Bench will evaluate the model across Tool Use, Memory Usage, Skill Usage, and Sub-agent Collaboration. To participate, request a private evaluation through our secure submission portal.
- Confidential by design: submissions go directly to the dedicated evaluation server; API credentials are never handled by GitHub Pages.
- Hidden-test integrity: benchmark queries, ground truth, and judge credentials remain isolated from submitted models.
- Reviewed results: each completed run is checked before its report is released to the submitting account.
- Expected turnaround: reports are normally returned within three business days, with one request allowed per account every seven days.
If you find MobilePA-Bench useful in your research, please cite our paper:
@article{zhu2026mobilepabench,
title = {MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks},
author = {Zhu, Yi and Wu, Xiongwei and Wang, Qiyi and Qu, Tingyu and Liu, Jiajun and Cao, Sihan and Chen, Long and Sun, Weigao and Zhu, Feida and Zhong, Yiran and Hoi, Steven},
journal = {arXiv preprint arXiv:2608.23035},
year = {2026},
eprint = {2608.23035},
archivePrefix = {arXiv},
primaryClass = {cs.AI}
}For questions, evaluation requests, or collaboration proposals, please open a GitHub issue.
Unless otherwise noted, this repository is licensed under the Apache License 2.0.