I tried to use it with https://github.com/hamr0/bareguard to check for attachments, PII, API keys but it performed bad and late compared to readme
What the POC found (measured, not quoted)
I ran the real base-tier ONNX against an independent corpus the model's own generator never produced — real repo files for type, freshly-generated real-format keys (throwaway, never live creds), and benign near-misses as negatives so the test could fail.
The riskiest assumption — the secrets head — fails the gate.
- Independent recall 0.46, precision 0.86 — vs the author's synthetic AP = 1.0.
- Root cause (confound-checked): it's a whole-blob classifier, not a span detector. The same sk_live_… key scores api_key 0.96 standalone but goes completely silent the moment it's line 2 of a 2-line .env, inside a code comment, or past the 1 KB window. Bare email/phone fire; an email in a normal sentence misses.
- That embedded shape is what a gate inspects (a credential is a minority of the bytes in tool I/O) — and the misses are exactly the cases a 5-line regex catches trivially. So it doesn't beat the deterministic floor; it underperforms it where it matters, and even as a "recall aid" the union doesn't help because its misses are a superset of the easy cases.
Secondary findings:
- File-type on real files: coarse 5/10 (config/YAML/Markdown/PDF all misread as code); code_lang/MIME don't generalize off the synthetic set (.js→scss, png→x-parquet). The synthetic 100% doesn't survive real files.
- Latency ~38 ms single-call, not the <6 ms headline (that's batch-amortized at batch_size=64; a gate pays single-call per tool call).
- Setup reality: the PyPI wheel ships no weights (deps are torch+safetensors, not onnxruntime); the README's from picotype import … is wrong (it's model.pico_type); the CLI softmaxes a multi-label head. I used the correct sigmoid — i.e. I was generous to it.
Honest gaps: I didn't test the pro tier (bigger/slower; unlikely to fix a structural limitation) or the onnxruntime-node JS path (moot given the accuracy result).
Verdict
Confirmed SEED/NOT-NOW — don't builministic pattern floor wins on thecase that matters. The only thing l bareagent-via-MCP type-routing angle, and even that is "nice-to-hales. Everything's written into §5.5 + §6 of the doc; the POC lives in scrpped).
I tried to use it with https://github.com/hamr0/bareguard to check for attachments, PII, API keys but it performed bad and late compared to readme
What the POC found (measured, not quoted)
I ran the real base-tier ONNX against an independent corpus the model's own generator never produced — real repo files for type, freshly-generated real-format keys (throwaway, never live creds), and benign near-misses as negatives so the test could fail.
The riskiest assumption — the secrets head — fails the gate.
Secondary findings:
Honest gaps: I didn't test the pro tier (bigger/slower; unlikely to fix a structural limitation) or the onnxruntime-node JS path (moot given the accuracy result).
Verdict
Confirmed SEED/NOT-NOW — don't builministic pattern floor wins on thecase that matters. The only thing l bareagent-via-MCP type-routing angle, and even that is "nice-to-hales. Everything's written into §5.5 + §6 of the doc; the POC lives in scrpped).