Skip to content

Commit 151d157

Browse files
nickklos10nickklos10
authored
Merge pull request #14 from nickklos10/codex/security-hardening
Harden SQL safety and complete open-source packaging
2 parents 5f69250 + b9cf461 commit 151d157

19 files changed

Lines changed: 1330 additions & 82 deletions

‎CHANGELOG.md‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# Changelog
2+
3+
All notable changes to QueryPilot are documented here. The format follows
4+
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project
5+
follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
6+
7+
## [Unreleased]
8+
9+
## [0.1.0] - 2026-04-28
10+
11+
Initial public release. Foundation and eval-driven harness shipped.
12+
13+
### Added — Safety engine
14+
15+
- sqlglot-based SQL validation with single-statement and SELECT-only
16+
enforcement, blocked-keyword filter, table/column allowlist + blocklist,
17+
Cartesian-join detection, automatic `LIMIT` injection and max-row capping,
18+
`SELECT *` policy, query fingerprinting, and risk-level scoring.
19+
- SQLite and PostgreSQL connectors with per-dialect read-only execution and
20+
query timeouts.
21+
22+
### Added — Audit & access control
23+
24+
- Structured audit trail: `QueryAuditRecord` with audit ID, original SQL,
25+
rewritten SQL, validation result, applied access policy, row count, and
26+
execution timing.
27+
- `InMemoryAuditSink` and `JSONLAuditSink` implementations.
28+
- Access-control policies: column allowlist + blocklist, row-filter injection
29+
ANDed into WHERE, post-execution column masking (redact / null / hash).
30+
31+
### Added — Runtimes
32+
33+
- FastAPI server runtime (`querypilot serve`) with `/health`, `/schema`,
34+
`/search-schema`, `/ask`, `/generate-sql`, `/validate-sql`, `/execute-sql`,
35+
`/evals/run`, and `/audit/recent` endpoints.
36+
- MCP server runtime (`querypilot mcp`) over stdio and Streamable HTTP with
37+
`ask_database`, `search_schema`, `validate_sql`, and `execute_sql` tools.
38+
39+
### Added — Generation
40+
41+
- `DemoSQLGenerator` (offline, deterministic) plus `OpenAISQLGenerator` and
42+
`AnthropicSQLGenerator` provider implementations with a validator-driven
43+
repair loop.
44+
- Tool-schema producers `as_openai_tools()` and `as_anthropic_tools()` for
45+
drop-in agent use.
46+
47+
### Added — Eval-driven harness
48+
49+
- `BenchmarkCase` / `BenchmarkSuite` Pydantic schema with thresholds,
50+
comparison config, and tags. YAML and JSON suite loaders with shared-config
51+
validation across multi-file suite directories.
52+
- Result-set comparator with column- and row-order normalization,
53+
abs-difference float tolerance, datetime ISO normalization, NaN handling,
54+
and structured row diffs.
55+
- `pipeline.run_case` per-case runner reusing existing `QueryPilot.generate_sql`,
56+
`validate_sql`, and `execute_sql`; per-stage `time.perf_counter` timings;
57+
failure classification across 12 categories.
58+
- Cost trackers for OpenAI and Anthropic (idempotent client wrap with
59+
restore), `NullCostTracker`, and a `MODEL_PRICING` table.
60+
- `run_suite` aggregator producing `SuiteReport` with pass / safety /
61+
correctness / repair rates, p50/p95 latency, token totals, $ estimate, tag
62+
rollups, failure breakdown, and threshold violations.
63+
- Stdlib JSON writer and screenshot-quality terminal renderer.
64+
- `querypilot eval run|replay|check|init` CLI with audit-log → regression-
65+
suite replay, baseline-vs-current regression detection, and a starter
66+
scaffold for `suites/` and `.eval/`.
67+
- Sample GitHub Actions workflow at `.github/workflows/eval.yml` running
68+
`eval run` + `eval check` on every PR.
69+
70+
[Unreleased]: https://github.com/nickklos10/QueryPilot/compare/v0.1.0...HEAD
71+
[0.1.0]: https://github.com/nickklos10/QueryPilot/releases/tag/v0.1.0

‎CONTRIBUTING.md‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# Contributing to QueryPilot
2+
3+
Thanks for considering a contribution. QueryPilot is an eval-driven SQL
4+
reliability toolkit for AI agents — every change is measured against an
5+
execution-truth eval suite, so the same standards apply to outside
6+
contributions as to maintainer commits.
7+
8+
## Local setup
9+
10+
```bash
11+
python3 -m venv .venv
12+
.venv/bin/pip install -e ".[dev,eval]"
13+
.venv/bin/pytest # full test suite
14+
.venv/bin/querypilot eval run \
15+
--suite suites/smoke.yaml \
16+
--generator demo \
17+
--report /tmp/eval-out.json # smoke harness end-to-end
18+
```
19+
20+
Optional extras for provider-specific work:
21+
22+
```bash
23+
.venv/bin/pip install -e ".[openai,anthropic,server,mcp]"
24+
```
25+
26+
## Workflow
27+
28+
1. **Branch off `main`.** Use a short, scope-prefixed name:
29+
- `eval/<slug>` for eval-harness changes
30+
- `launch/<slug>` for distribution / launch-readiness changes
31+
- `fix/<slug>` for bug fixes
32+
- `feat/<slug>` for new features
33+
2. **One PR per logical change.** Keep changes focused — multi-PR sequences
34+
are preferred over one large PR. PR-N+1 doesn't start until PR-N is
35+
merged.
36+
3. **Tests are required.** New behavior needs a unit test; new validator
37+
guards need a safety-suite case; user-facing CLI changes need a CLI test.
38+
4. **`/code-review` before merge.** Every PR gets a multi-agent code review
39+
pass. Treat issues that score ≥ 80 as blocking; bundle below-threshold
40+
real concerns as a follow-up commit.
41+
5. **Don't break the eval baseline.** CI runs `querypilot eval check` against
42+
`.eval/baseline.json`. If your change deliberately changes pass/safety/
43+
correctness rates, regenerate the baseline in the same PR and explain why
44+
in the description.
45+
46+
## Code style
47+
48+
- Python 3.11+, full type hints, `from __future__ import annotations` at the
49+
top of new modules.
50+
- Pydantic v2 for serialized data shapes. Pure dataclasses or plain classes
51+
for internal-only data.
52+
- No new mandatory dependencies. New integrations live behind a
53+
`[project.optional-dependencies]` extra (see `[eval]`, `[openai]`,
54+
`[anthropic]`).
55+
- Keep the core (`src/querypilot/core/client.py`) clean. New eval, replay, or
56+
adapter code should compose existing public methods, not instrument the
57+
core.
58+
59+
## Reporting security issues
60+
61+
Don't open public issues for security vulnerabilities. Email
62+
`nklos@inceptaanalytics.ai` instead. See `SECURITY.md` for the threat model
63+
and disclosure process.
64+
65+
## License
66+
67+
By contributing, you agree your contributions are licensed under the MIT
68+
License (see `LICENSE`).

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 Nicholas Klos
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,10 @@
11
# QueryPilot
22

3+
[![PyPI](https://img.shields.io/pypi/v/querypilot.svg)](https://pypi.org/project/querypilot/)
4+
[![Python](https://img.shields.io/pypi/pyversions/querypilot.svg)](https://pypi.org/project/querypilot/)
5+
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
6+
[![CI](https://github.com/nickklos10/QueryPilot/actions/workflows/eval.yml/badge.svg)](https://github.com/nickklos10/QueryPilot/actions/workflows/eval.yml)
7+
38
Eval-driven SQL reliability for AI agents.
49

510
QueryPilot helps agents safely generate, validate, repair, execute, and regression-test SQL against real fixture databases.
@@ -306,6 +311,12 @@ QueryPilot validates SQL before execution with:
306311
- query fingerprints
307312
- risk levels: `low`, `medium`, `high`, `critical`
308313

314+
For PostgreSQL production use, connect QueryPilot with a dedicated
315+
least-privilege role that has only the required schema `USAGE` and table
316+
`SELECT` grants. QueryPilot requests a read-only transaction and applies a
317+
statement timeout, but application validation is not a replacement for
318+
database permissions.
319+
309320
Example:
310321

311322
```python

0 commit comments

Comments
 (0)