This maps the developer needs in research/developer-needs.md against what QueryPilot ships in main (v0.1.1) and ranks the gaps. Last updated 2026-10-20.
Impact is how much the change helps an engineer adopting QueryPilot, or how much it lifts SQL quality. Effort is S (a day or less), M (a few days) or L (a week or more). Items marked this PR are the ones implemented on feat/agent-dx.
PR #25 fixes the validator scope (CTEs, select aliases, subqueries, UNION), applies masking through column lineage, caps non-literal LIMITs and opens SQLite read-only. On main, 39 of the 65 SaaSPulse correctness golds fail QueryPilot's own validator; with PR #25 all 65 pass. Everything below assumes PR #25 has landed, and nothing here touches validation/validator.py or connectors/sqlite.py.
| # | Gap in QueryPilot today | Need it answers | Impact | Effort | Evidence | Status |
|---|---|---|---|---|---|---|
| 1 | Only SQLite and Postgres are supported. The dialect isn't inferred from the URL: QueryPilot.connect("postgresql://…") silently builds a SQLite connector. Postgres sees only the default schema, and views are ignored. |
Missing connectors, real-world connectivity | High | M | WrenAI connector requests, Vanna #285, LangChain #18742 | this PR |
| 2 | There are no framework adapters. The de-facto SQL toolkit (langchain-community) is sunset and its SQL bugs were closed unfixed. Only Anthropic has a tool-call dispatcher, and it raises on bad SQL. |
Drop-in tools for LangChain, LlamaIndex, OpenAI Agents and Claude Agent SDK | High | M | langchain-community #674, Vanna #488, deepagents #138 | this PR |
| 3 | Agent tools can't see the schema: there is no list_tables or describe_table. Errors come back as exceptions or 1.5 KB validation dumps. Results carry no truncated flag, and MCP tools have no readOnlyHint annotations. |
Token-efficient schema discovery, explicit bounded results | High | S–M | Supabase MCP #52, DBHub #404, Metabase #82850 | this PR |
| 4 | Repair feedback is weak. Errors are bare ("Unknown column: nme") with no suggestions, and parse errors include ANSI escape codes. Database errors and empty results are never repaired, and the default allows only one repair. | Catalog-grounded, actionable errors; repair that converges | High (quality) | M | CHESS repair −6.80 EX when removed (arXiv 2405.16755), CHASE fixer +3.78 (2410.01943), Vanna #279 | this PR |
| 5 | Schema context is thin: table(col TYPE, …) for every table, with no keys, comments, sample values or business hints. The schema is also re-introspected on every call. |
Schema grounding, value grounding, domain hints | High (quality) | M | BIRD evidence +13–20 EX (2305.03111), 3 sample values +2.6–3.2 (2305.11853), FKs +0.6–2.9 (2308.15363) | this PR |
| 6 | Large schemas get no pruning; the whole schema goes into every prompt. | Large-schema handling | Medium–High | M | Prune only past a size threshold (2408.07702), MAC-SQL selector +2.1 (2312.11242), LlamaIndex #6886 | this PR (threshold-gated, keeps FK bridge tables) |
| 7 | The first five minutes are hard. The README has no pip install querypilot, the demo DB isn't shipped, and Quick Demo starts with evals. There's no querypilot ask/schema command, and serve/mcp can't pick an LLM, so ask_database over MCP is demo-only. |
Painless install and setup | High | S–M | Vanna #985, python-sdk #478, dbt MCP troubleshooting | this PR |
| 8 | The MCP registry entry (uvx querypilot mcp) installs without the mcp SDK. |
Zero-friction MCP install | Medium | S | python-sdk #478 | this PR |
| 9 | Dangerous-function blocklists are Postgres-only (no DuckDB file readers, MySQL LOAD_FILE/SLEEP, or MSSQL xp_cmdshell/OPENROWSET). |
Read-only that holds across engines | Medium | S | Postgres MCP Pro #178, WrenAI #2679 | this PR (ships with #1) |
| 10 | CI doesn't gate SaaSPulse golds or the safety suite. | Regression evals in CI | Medium | S | WrenAI eval epic; PR #25 adds a gold-validation test | Partly in PR #25 |
| 11 | No multi-candidate generation with result voting, and no anomaly guard (fan-out, empty, duplicates). | Quality | Medium | M | self-consistency +0.4–2.7, ModularSQL +1.9 (research-quality) | Later |
| 12 | No few-shot retrieval from verified question→SQL pairs (audit log or approvals). | Quality | Medium | M | DAIL-SQL, MARLO +2.9 | Later |
| 13 | No warehouse cost guards (BigQuery dry-run / maximum_bytes_billed, Snowflake statement timeout) and no SSH tunnel or multi-database server. |
Cost guards, connectivity | Medium | M | MCP Toolbox BigQuery, DBHub SSL/SSH issues | Later |
| 14 | No authenticated streamable-HTTP deployment story. | Remote MCP | Low–Medium | M | research-mcp #8 | Later |
| 15 | The eval comparator ignores whether the gold query has ORDER BY, and expected_columns is never checked. |
Comparator that doesn't lie | Low–Medium | S | Set-EX overstates by 3.4–6.8 pp | Later |
- Connect from any URL.
QueryPilot.connect(url)infers the dialect, and a generic SQLAlchemy connector covers DuckDB, MySQL/MariaDB, SQL Server, Snowflake, BigQuery, Redshift, Oracle, Trino and Databricks. Introspection covers PKs, FKs, comments, views and non-default schemas, and the schema is cached. Each engine uses its native read-only mode where one exists. Dialect-aware blocked functions ship here too. - Agent toolkit. A single
QueryPilotToolkitdefineslist_tables,describe_table,search_schema,run_sql/execute_sql,validate_sqlandask_database. It returns structured errors instead of raising and reportstruncated. It is exposed as OpenAI (Chat and Responses), Anthropic, MCP (withreadOnlyHint), LangChain (withsql_db_*aliases), LlamaIndex, OpenAI Agents SDK and Claude Agent SDK tools. - Repair that converges. Errors suggest likely names ("did you mean
customer_name?"), list valid columns and strip ANSI codes. Database errors and empty results also go back to the model. The default is 3 attempts. - Grounded schema context. The prompt includes PKs, FKs, comments, up to 3 sample values for text columns, and user-supplied
hints(a glossary or business rules). Above a token budget, only relevant tables are sent, plus the FK bridge tables that connect them. - First five minutes.
pip install querypilot, thenquerypilot ask "…" --db URL,querypilot schema --db URL, andquerypilot demo, which uses a bundled demo DB.serve/mcptake--generator/--model, andquerypilot mcp-config <client>prints the client config. server.json is fixed, and the README is reordered so the first query comes first. - Measure. SaaSPulse is run before and after with the same model.