Please note that before your evaluation process, when Docker loads the databases, errors may occasionally occur due to environment inconsistency (these will not terminate the process but will appear in the Docker logs). As a result, some databases may fail to load properly, leading to empty databases. This will cause the evaluation results to be abnormally low.
👉 Therefore, we strongly recommend checking the Docker logs for any errors before running the evaluation and verifying that all databases have been successfully loaded.
👉 We have updated the Submission Guidelines, where the customized agent scaffolds are supported. Please feel free to take a look at our detailed submission guidelines here.
-
[2025-10-21] 🚀 Docker update: We added the docker for Full DB Env. And we pushed 3 docker images (Base/Full DB Env and the evaluation environment for both
a-Interactandc-Interact) to Docker Hub to facilitate the environment setup. No need to download the DB dumps and build the images manually! -
[2025-10-08] 📝 Our Bird-Interact paper is now publicly available!
It presents the full details, methodology, and evaluation of our interactive text-to-SQL benchmark.
👉 Check it out and know more about the ideas behind BIRD-Interact. -
[2025-08-26] 🚀 We're excited to announce the release of the BIRD-Interact-Full (600) set!
It's a tough one — the best LLMs are only achieving a 16.33% success rate, with just 10.0% on thec-interactanda-interactportions.
👉 For more details, please visit our project website. -
[2025-08-26] 📬 We'll be sending the Ground Truth & Test cases to our mailing list this week.
If you want early access, please send an email as instructed on the site for an automatic download. -
[2025-08-26] 💾 On another note, we've also released a SQLite version of LiveSQLBench-Lite for easier local research.
The full LiveSQLBench-Base and -Large versions are coming soon! -
[2025-08-22] Bug Fix: In Bird-Interact-Agent code, we fixed a bug that when evaluating phase-2 SQL, the stored phase-1 SQL cannot be executed successfully, leading to a lower success rate of Phase-2. This bug only affects those tasks where phase1 sql does some operations on the database, e.g. CREATE table, etc.
BIRD-INTERACT, an interactive text-to-SQL benchmark, re-imagines Text-to-SQL evaluation via lens of dynamic interactions. The environment blends a hierarchical knowledge base, database documentation and a function-driven user simulator to recreate authentic enterprise environments across full CRUD operations. It offers two rigorous test modes: (1) passive Conversational Interaction and (2) active Agentic Interaction, spanning 600 annotated tasks including Business Intelligence (BI), CRUD operations and etc., each guarded by executable test cases. Typical evaluations trigger 1,968-5,496 interaction turns between model and user simulator, while state-of-the-art reasoning models currently solve only ≈24% and ≈18% of tasks, underscoring the benchmark's challenge.
BIRD-INTERACT supports two evaluation modes as mentioned above:
- c-Interact: Conversational Interaction which is a passive mode and the workflow is fixed. The code and detailed information can be found in
bird_interact_conv. - a-Interact: Agentic Interaction which is an embodied active mode where the workflow is dynamic and led by models. The code and detailed information can be found in
bird_interact_agent.
We are releasing a lite version of BIRD-INTERACT, bird-interact-lite-exp, which includes 270 high-quality real-world tasks specifically for PostgreSQL. This is a good starting point for quick experimentation.
The full version of BIRD-INTERACT, bird-interact-full, is a comprehensive benchmark that includes 600 tasks for PostgreSQL. It covers a wide range of SQL operations and user queries. The full version is coming soon.
| Rank | Model Name | Normalized Reward | Avg Cost (USD)/Task | Level |
|---|---|---|---|---|
| 1 | Gemini-2.5-Pro | 20.92 | $0.04 | 🏆 Excellent Chat |
| 2 | O3-Mini | 20.27 | $0.07 | 🏆 Excellent Chat |
| 3 | Claude-Sonnet-4 | 18.35 | $0.29 | 💎 Good Chat |
| 4 | Qwen-3-Coder-480B | 17.75 | $0.11 | 💎 Good Chat |
| 5 | Deepseek-Chat-V3.1 | 15.15 | $0.12 | ✨ Standard |
| 6 | Claude-Sonnet-3.7 | 13.87 | $0.29 | ✨ Standard |
| 7 | GPT-5 | 12.58 | $0.08 | ⚪ Basic |
| Rank | Model Name | Normalized Reward | Avg Cost (USD)/Task | Level |
|---|---|---|---|---|
| 1 | GPT-5 | 25.52 | $0.24 | 🏆 Excellent Interaction |
| 2 | Claude-Sonnet-4 | 23.28 | $0.51 | 🏆 Excellent Interaction |
| 3 | Claude-Sonnet-3.7 | 17.45 | $0.60 | 💎 Good Interaction |
| 4 | Gemini-2.5-Pro | 17.33 | $0.22 | 💎 Good Interaction |
| 5 | O3-Mini | 16.43 | $0.06 | ✨ Standard |
| 6 | Deepseek-Chat-V3.1 | 13.47 | $0.06 | ✨ Standard |
| 7 | Qwen-3-Coder-480B | 10.58 | $0.07 | ⚪ Basic |
* Budget Parameters: Starting Budget/User Patience Budget, measured by our virtual currency bird-coins
. Refer to bird_interact_agent/README.md for more details.
Interaction-Time Scaling (ITS) refers to a model's ability to continuously increase its end performance through multi-turn interactions. When this interactive performance surpasses the model's idealized single-turn performance on a fully specified, unambiguous task, we say it satisfies the ITS law. As user patience grows and interaction turns accumulate, performance keeps improving, demonstrating that the model can sustain effective communication over extended dialogue. Currently, we only find claude-3-7-sonnet satisfies the ITS law.
- Run Docker containers for bird-interact-lite database, bird-interact-full database, and evaluation environment:
| If you just want to evaluate on bird-interact-lite, you could comment out the postgresql_full service in docker-compose.yml to speed up the environment setup.
Start the environment by running:
cd env
docker compose pull
docker compose up -dWait for several minutes for database initialization.
This runs 3 containers using prebuilt images from Docker Hub:
bird_interact_postgresql: PostgreSQL database for bird-interact-litebird_interact_postgresql_full: PostgreSQL database for bird-interact-fullbird_interact_eval: Evaluation environment for botha-Interactandc-Interact.
Now, you could start the evaluation environment by executing the following command:
docker compose exec bird_interact_eval bash-
(Optional) Build the environment manually (if you want to build the images from scratch):
- Downdload the database dumps
- bird-interact-lite. Unzip and rename it as
env/postgre_table_dumps. - bird-interact-full. Unzip and rename it as
env/postgre_table_dumps_full.
- bird-interact-lite. Unzip and rename it as
- Build the environment manually by running
docker-compose.build.yml.cd env/ docker compose -f docker-compose.build.yml build docker compose -f docker-compose.build.yml up -d
- Downdload the database dumps
-
(Recommended) Check the database containers are built and running successfully.
-
Print the container build logs to ensure that the databases are built successfully without errors:
docker logs bird_interact_postgresql > build_bird_interact_postgresql.log 2>&1 docker logs bird_interact_postgresql_full > build_bird_interact_postgresql_full.log 2>&1
If errors occur,
"Errors occurred during import:"will be printed in the log files. -
Check if the database containers are in good shape.
Use our provided Python script to verify database metadata:
docker compose exec bird_interact_eval bash cd /app/env python check_db_metadata.py --host bird_interact_postgresql python check_db_metadata.py --host bird_interact_postgresql_full
Expected results:
- bird-interact-lite:
- 📈 Total Databases: 18
- 📋 Total Tables: 175
- 🔢 Total Columns: 2286
- 📈 Avg Rows per Table: 1,038.48
- 💾 Total Size: 207.15 MB (around)
- bird-interact-full:
- 📈 Total Databases: 22
- 📋 Total Tables: 244
- 🔢 Total Columns: 2011
- 📈 Avg Rows per Table: 1,121.19
- 💾 Total Size: 272.00 MB (around)
- bird-interact-lite:
-
Database: The complete PostgreSQL database can be download from the Google Drive. Check the Quick Eval section for more details.
-
data: Each data instance contain the following main parts:
selected_database: The name of the database.query: The unambiguous user query.amb_user_query: The user query with injected ambiguities.user_query_ambiguity: The ambiguities injected into the user query.non_critical_ambiguity: The non-critical ambiguities like order, limit, etc.knowledge_ambiguity: The ambiguities created by masked external knowledges.sol_sql: The ground truth SQL solution.preprocess_sql: SQL queries to run before executing the solution or prediction.clean_up_sql: SQL queries to run after the test cases to revert any changes made to the database.test_cases: A set of test cases to validate the predicted corrected SQL.follow_up: The labeled follow up questions.external_knowledge: The external knowledge related to the specific task.
-
evaluation: The evaluation code is available in the
./evaluationdirectory. -
Curated by: BIRD Team & Google Cloud
-
License: cc-by-sa-4.0
-
HuggingFace Dataset Card: bird-interact-lite
To avoid data leakage by auto-crawling, we do not include GT solution sqls and test cases along with data.
please email [email protected] with the tag [bird-interact-lite GT&Test Cases] in title for full set, which will be sent automatically.
.
├── LICENSE
├── README.md
├── bird_interact_conv
│ ├── ...
│ └── README.md
├── bird_interact_agent
│ ├── ...
│ └── README.md
├── evaluation
│ ├── docker-compose.yml
│ ├── env
│ ├── postgre_table_dumps
│ ├── run
│ └── src
├── materials
│ ├── ...
└── requirements.txt
The details about running a-interact can be found in ./bird_interact_agent/README.md; and c-interact can be found in ./bird_interact_conv/README.md.
- Release lite version, bird-interact-lite (270).
- Release conversational version, bird-interact-conv.
- Release agent version, bird-interact-agent.
- Release Full bird-interact-full (600).
- SFT / RL an User Simulator
We would like to express our sincere gratitude to Irina Saparina, Mohammadreza Pourreza, Mehdi Bouzouina, Hailong Li, Jiatong Shi, and Professor Shinji Watanabe for their fruitful discussions and valuable insights that helped improve this project.
BIRD Team & Google Cloud
