Add DeepSeek model client + CLINC OOS classification support - #16
Open
purva-8 wants to merge 3 commits into
Open
Add DeepSeek model client + CLINC OOS classification support#16purva-8 wants to merge 3 commits into
purva-8 wants to merge 3 commits into
Conversation
- src/models/deepseek_client.py: DeepSeek chat client via the OpenAI SDK
pointed at api.deepseek.com (OpenAI-compatible API), reusing the
retry/pricing pattern from OpenAIClient.
- Register "deepseek" in MODEL_REGISTRY with default pricing for
deepseek-v4-flash and deepseek-v4-pro (cache-miss input rate, deepseek.ai
pricing as of Aug 2026).
- src/dataset/clinc_oos.py + src/tasks/classification_task.py: net-new
Classify-task support (CLINC OOS intent classification, 151 labels incl.
out-of-scope), scored via the existing compute_exact_match metric with a
label-normalizing Task.normalize().
- config/experiments.yaml: qmsum_baseline_deepseekv4{flash,pro} at the same
num_examples=281/seed=40 as the existing GPT/Gemini QMSum entries, plus
classify_clinc_baseline_deepseekv4{flash,pro} (num_examples/seed here are
placeholders pending confirmation against whatever sample size produced
the other models' published Classify numbers).
Compress (FinanceBench) intentionally left out of this PR — the dataset
loader's PDF path is hardcoded to a contributor's local machine
(src/dataset/financebench.py DEFAULT_PDF_DIR), so it's unclear whether the
existing Compress numbers were scored against full PDFs or the
evidence-text fallback. Extract (LongExtractBench) is also out of scope
for now — no dataset loader, task, or field-level metric exists in this
repo yet.
…s) support
- src/dataset/longextractbench.py: loader for micro1-inc/longextract-bench-50
(a raw-file HF dataset repo, one folder per example with document.pdf +
schema.json + ground_truth.json). Uses huggingface_hub.snapshot_download +
PyMuPDF, mirroring dataset/financebench.py's PDF handling. Verified against
the live repo: 50 example folders, matching num_examples=50 in the new
experiment entries (full set, no sampling needed).
- src/tasks/extraction_task.py: builds a schema-in-system-prompt extraction
prompt, strips markdown fences models sometimes wrap JSON in.
- src/metrics/field_extraction.py: field-level precision/recall/F1 + parse
rate, scored by flattening predicted/gold JSON into dotted-path pairs and
comparing as sets. One function per metric name to match the existing
one-name-per-key convention (see metrics/retrieval.py) — a single function
returning multiple keys would silently drop all but one under how
evaluation/runner.py picks a key out of each metric's return dict.
This is a first-pass scorer: exact string match after light normalization,
no fuzzy numeric/date tolerance, no schema validation.
- src/dataset/financebench.py: DEFAULT_PDF_DIR was hardcoded to a
contributor's local machine path. Now reads FINANCEBENCH_PDF_DIR (or
defaults to data/financebench-pdfs/ under the repo root) and documents the
PDF-vs-fallback context-source risk directly in the comment.
- config/experiments.yaml: financebench_baseline_deepseekv4{flash,pro}
(Compress, n=150/seed=42, matching the existing GPT-5.2 FinanceBench
entries) and extract_longextractbench_deepseekv4{flash,pro} (Extract,
n=50 = full set).
Verified: registry resolves all new dataset/task/metric names; extraction
task + field metrics tested against synthetic perfect/partial/unparseable
predictions (1.0 / 0.667 / 0.0 as expected); live HF API call confirms the
LongExtractBench repo has exactly 50 example folders. Not yet run against a
live DeepSeek key — blocked on account balance (see PR discussion).
- src/models/openrouter_client.py: new provider routing DeepSeek (and any
other OpenRouter-hosted model) through openrouter.ai's OpenAI-compatible
API. Disables upstream reasoning mode by default — confirmed live that
DeepSeek V4 Flash/Pro on OpenRouter default to an extended-thinking mode
that burns completion tokens on a hidden `reasoning` field before (or
instead of, if max_tokens runs out first) visible `content`. At
max_tokens=10 with reasoning left on, content came back null with
finish_reason="length" — silently breaks any low-max_tokens task like
Classify. Reports OpenRouter's actual per-call cost from the response
usage block when available, falling back to static PricingInfo otherwise.
- src/models/base.py: pricing entries for the OpenRouter-routed model ids.
- Fixed a real, pre-existing bug affecting every non-Summarization baseline
task: evaluation/runner.py unconditionally passes model_mode into every
task's build_prompt(), but only SummarizationTask popped it before
forwarding **kwargs to GenerationInput() (which has no such field).
Confirmed live against RAGTask — `financebench_baseline_gpt52` and any
other rag_qa/retrieval_task/classification/extraction experiment would
raise "GenerationInput.__init__() got an unexpected keyword argument
'model_mode'" the moment it actually ran. Fixed by popping model_mode in
RAGTask, RetrievalTask, ClassificationTask, and ExtractionTask.
- src/dataset/cuad.py: CUAD (Contract Understanding Atticus Dataset) loader,
replacing LongExtractBench for the Extract task's DeepSeek entries — CUAD
is the dataset actually used for Extract on the existing Grok/OpenAI/Gemini
benchmark pages (F1 metric on contract clause spans). Loaded from
chenghao/cuad_qa, a SQuAD-format Parquet mirror (the official
theatticusproject/cuad-qa loader script no longer loads under current
`datasets` versions, which dropped script-based loaders).
- config/experiments.yaml: extract_cuad_baseline_deepseekv4{flash,pro}
(rag_qa task, exact_match+f1, n=150/seed=42 matching FinanceBench's
pattern). LongExtractBench entries are left in place but not the active
path for Extract — see PR discussion for why.
Why the CUAD switch: LongExtractBench documents run ~250K input tokens with
schemas that need far more than 12K output tokens to fully extract as valid
JSON — piloted at 8 examples, hit 0% parse rate even at 12K max_tokens, one
call took 33 minutes and $0.10. CUAD's ~11K-char contexts are 20x cheaper
and 100x+ faster per call (8.2s vs 6.9min avg in pilot testing) and, more
importantly, match what actually produced the other models' Extract numbers.
Verified: full pilot sweep (8 examples/model) now passes cleanly across all
four tasks — Summarize (QMSum), Classify (CLINC OOS), Compress
(FinanceBench), Extract (CUAD) — for both deepseek-v4-flash and
deepseek-v4-pro via OpenRouter, with real costs/latencies/scores recorded
(e.g. CUAD F1 0.305 for V4 Flash, QMSum ROUGE-L 0.192, CLINC accuracy 0.75
on n=8 pilots). LongExtractBench's dataset/task/metric code is left in the
repo (dataset structurally verified, scoring logic unit-tested) in case a
future pass wants to solve the token-budget problem properly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
All four ScaleDown benchmark-page tasks are now working end-to-end for DeepSeek V4 Flash/Pro, verified with real API calls via a funded OpenRouter key.
What's added
src/models/deepseek_client.py+src/models/openrouter_client.py: two ways to reach DeepSeek — direct API (deepseek:provider) and via OpenRouter (openrouter:provider, currently the working path since the direct-API key hit402 Insufficient Balance). OpenRouter routing disables upstream reasoning mode by default — confirmed live that DeepSeek V4 Flash/Pro on OpenRouter default to an extended-thinking mode that burns completion tokens on a hiddenreasoningfield before/instead of visiblecontent. Atmax_tokens=10with reasoning left on,contentcame backnullwithfinish_reason="length"— this silently breaks any low-max_tokenstask like Classify if not caught.evaluation/runner.pyunconditionally passesmodel_modeinto every task'sbuild_prompt(), but onlySummarizationTaskpopped it before forwarding toGenerationInput()(which has no such field). Confirmed live againstRAGTask— this meansfinancebench_baseline_gpt52and any otherrag_qa/retrieval_taskexperiment would crash the moment it actually ran. Fixed inRAGTask,RetrievalTask,ClassificationTask,ExtractionTask.src/dataset/clinc_oos.py+src/tasks/classification_task.py: net-new Classify support (151 labels incl. out-of-scope), scored via existingcompute_exact_match.src/dataset/cuad.py: Extract now uses CUAD, not LongExtractBench — see "Why the switch" below. Reuses the existingrag_qatask +exact_match/f1metrics, same pattern as FinanceBench.src/dataset/longextractbench.py+src/tasks/extraction_task.py+src/metrics/field_extraction.py: LongExtractBench support is still in the repo (dataset structurally verified against the live HF repo, task/metric logic unit-tested against synthetic fixtures) but not the active Extract path — parked for a future pass. See below.src/dataset/financebench.py:DEFAULT_PDF_DIRwas hardcoded to a contributor's local machine path. Now readsFINANCEBENCH_PDF_DIRenv var with a documented fallback.config/experiments.yaml: 10 new experiment entries across all four tasks × 2 models.Why CUAD instead of LongExtractBench for Extract
LongExtractBench documents run ~250K input tokens with schemas needing far more than 12K output tokens to fully extract as valid JSON. Piloted at 8 examples: 0% JSON parse rate even at 12K max_tokens, one single call took 33 minutes and $0.10. CUAD's ~11K-char contexts are ~20x cheaper and ~50x faster per call (8.2s vs 6.9min avg in pilot testing), and — more importantly — CUAD is what actually produced the Extract numbers on the existing Grok/OpenAI/Gemini pages, so this keeps DeepSeek comparable rather than introducing a different benchmark.
Verified (all with real API calls against a funded key)
DeepSeekClient/OpenRouterClientauth + model routing confirmed live.Still open
num_examples/seedare placeholders — nothing in this repo records the sample size used for the other models' published Classify/Extract numbers.FINANCEBENCH_PDF_DIRset (falls back toevidence_text_full_page).Test plan
py_compileon all new/changed files🧙 Built with WozCode