New Benchmark: How Frontier LLMs Perform as Security Guardrails

[akarlaraytu]

Hey everyone 👋

I just published the results of a systematic benchmark I ran:
Testing GPT-4.1, GPT-4o, Claude Sonnet 4, and Gemini 2.0 Flash as policy enforcement layers against 22 adversarial attack scenarios.

The short version:

• Every model has exploitable failure modes
• Best LLM (Claude Sonnet 4) blocked 86% of attacks; worst (GPT-4.1) blocked 45%
• 3 attack patterns broke every single model; consistently, across all runs
• All models scored 100% on legitimate transactions, so they understand the policy fine; they just can't defend it under adversarial pressure

The 3 universal bypasses:

1. Gradual Role Escalation - multi-turn context buildup + spoofed system message
2. Multi-Format Context Flood - same lie in HTTP headers, JSON, XML, and system context simultaneously
3. Invisible Character Injection - a zero-width space before "ADMIN" that LLMs can't catch but deterministic string comparison trivially rejects

CSL-Core was included as the deterministic baseline and held across all 22 scenarios at ~0.84ms per decision (vs 400-900ms for LLMs). The article goes into the architectural reasons why, and where each approach makes sense.

Full write-up with charts, heatmaps, and detailed analysis:
I Benchmarked 4 Frontier LLMs as Security Guardrails. None of Them Passed.

Benchmark code is fully open source:
https://github.com/Chimera-Protocol/csl-core/tree/main/benchmarks

Would love to hear your thoughts, especially if you've run into similar issues using LLMs for access control or policy enforcement in production. And if you want to run the benchmark against other models or with different policy domains, PRs are welcome.