Skip to content
eval

Add querypilot proxy: Postgres wire-protocol proxy that enforces QueryPilot policies #43

Sign in to view logs

Add querypilot proxy: Postgres wire-protocol proxy that enforces QueryPilot policies

Add querypilot proxy: Postgres wire-protocol proxy that enforces QueryPilot policies #43

Workflow file for this run

.github/workflows/eval.yml at bfcc5c5
name: eval
on:
pull_request:
push:
branches: [main]
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
# Pinned exactly: ruff's default rule set changes between minor releases,
# so bump this deliberately and fix the new findings in the same PR.
- name: Install ruff
run: pip install ruff==0.17.1
- name: Ruff check
run: ruff check src tests
eval:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ["3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: pip
- name: Install package
run: pip install -e ".[dev,eval]"
- name: Seed demo fixture
run: python tests/fixtures/seed_demo.py
- name: Run unit tests
run: pytest
- name: Validate SaaSPulse golds and safety cases
run: python benchmarks/validate_golds.py
- name: Quickstart smoke
run: |
querypilot demo --path "$RUNNER_TEMP/demo.db"
querypilot sql --db "sqlite:///$RUNNER_TEMP/demo.db" "SELECT segment, COUNT(*) AS n FROM customers GROUP BY segment"
querypilot schema --db "sqlite:///$RUNNER_TEMP/demo.db" customers
- name: Run smoke eval suite
run: |
querypilot eval run \
--suite suites/smoke.yaml \
--generator demo \
--report eval-out.json \
--no-color
- name: Check eval report against baseline
run: |
querypilot eval check \
--report eval-out.json \
--baseline .eval/baseline.json \
--threshold 0.9 \
--require-safety 1.0
- name: Upload eval report
if: always()
uses: actions/upload-artifact@v4
with:
name: eval-report-py${{ matrix.python-version }}
path: eval-out.json