Add more benchmark tasks (20 → 50 tasks)

[jackjin1997]

Goal

Expand from 10 to 50 benchmark tasks for more statistically meaningful results.

Proposed new tasks

Code (5 more)

• Refactor a class hierarchy to use composition over inheritance
• Add type hints to an untyped Python module
• Fix a race condition in async code
• Implement a caching decorator with TTL
• Convert callbacks to async/await

Data (5 more)

• Clean and merge two messy CSVs
• Build a simple ML pipeline (train/eval/predict)
• Visualize time series with anomaly highlights
• Parse and analyze a web server access log
• Generate a PDF report from structured data

Research (3 more)

• Compare 3 database options for a specific use case
• Write a threat model for a web application
• Summarize and critique a technical RFC

Tool Use (4 more)

• Set up a GitHub Actions workflow
• Create and configure a Docker Compose stack
• Automate a Slack notification pipeline
• Build a simple MCP server

Multi-step (3 more)

• Full PR workflow: branch → code → test → PR
• Debug production incident from logs to fix
• Migrate a project from JavaScript to TypeScript

How to contribute

See task authoring guide. Each task is a single YAML file.

[IlyasFardaouix]

I think the likely root cause of the issue is that the existing benchmark tasks are hardcoded and not easily extensible. The addition of 40 new tasks will likely require significant changes to the underlying architecture. I'd suggest reviewing the agentbench.adapters.registry module, specifically the register_adapter function, to see how new adapters and tasks are currently being added. This might give us some insight into how to make the process more scalable.

[jackjin1997]

Thanks for the input! To clarify — tasks are just standalone YAML files in tasks/, completely independent of the adapter registry. Adding a new task requires zero changes to existing code:

tasks/code/my-new-task.yaml   # task definition
tasks/fixtures/my-new-task/   # fixture files (test cases, starter code, etc.)

So scaling from 10 → 50 tasks is straightforward. The task authoring guide covers the full spec.

If you're interested in contributing tasks, the easiest starting point would be the Code domain — tasks like "fix a race condition in async code" or "implement a caching decorator" are self-contained and easy to auto-evaluate with pytest.

PRs welcome for any of the tasks listed above!

[IlyasFardaouix]

I think we can rule out the adapter registry as a bottleneck, given that tasks are independent YAML files. However, I'd like to propose a potential issue with the existing task loading mechanism: the tasks.load_tasks() function uses a simple glob to discover tasks, which might become inefficient with 50 tasks. Have you considered switching to a more robust task discovery mechanism, such as using a specific directory structure or a database to store task metadata?