Add GEMINI_SAFETY_THRESHOLD env to pass safetySettings through to Gemini's OpenAI-compat endpoint

[thekoma]

Problem

Running the Gemini compression provider against a real-world corpus of Claude Code transcripts (Italian + English, regular software-engineering discourse — includes the occasional curse word, frustrated venting, security-research terms) hits a silent-filter failure mode that's hard to diagnose without grepping logs.

Repro signature:

[agentmemory] error Compression failed {
  "obsId":"obs_…",
  "error":"gemini returned unexpected response: {\"choices\":[{\"finish_reason\":\"stop\",\"index\":0,\"message\":{\"role\":\"assistant\"}}],\"created\":…,\"id\":\"…\",\"model\":\"gemini-2.5-flash\",\"object\":\"chat.completion\",\"usage\":{…}}"
}

What's happening: Gemini's safety classifier is suppressing the assistant message content but the OpenAI-compat endpoint still returns 200 OK with finish_reason: "stop" and an empty message object. agentmemory's provider correctly flags it as unexpected response, but after ~7 such failures the resilient-provider circuit breaker opens — and the next ~900 observations get rejected without ever calling Gemini.

In my case the trigger was a backfill of ~12k observations across 11 historical sessions, and the cascade was clearly visible in kubectl logs:

914 "error":"circuit_breaker_open"
  7 "error":"gemini returned unexpected response: …"

The 7 "real" failures snowball into ~99% of the run getting dropped from compression. This is the same failure shape PR #845 describes ("one provider outage takes the whole memory hot-path down").

Why this is orthogonal to PR #845

#845 lands a graceful fallback: when Gemini fails, build a synthetic deterministic summary instead. That's the right move and I want it. But there's a category of users for whom synthetic summary is a downgrade they didn't ask for — they configured Gemini precisely because they wanted LLM-quality narratives, and the trigger isn't a Gemini outage, it's Gemini's safety filter on benign content (a transcript that happens to contain "I'm so frustrated with this f***ing config", a security review mentioning malware, a session about content moderation tooling, etc.).

For those users, the right answer is: don't get blocked. Pass safetySettings and keep using the LLM they paid for. #845 then becomes a true safety net for real outages (429s, network errors, malformed responses), not a workaround for over-zealous defaults.

Workaround I'm using today (and why it's incomplete)

Switching to GEMINI_MODEL=gemini-3-flash-preview made the issue disappear on the same corpus — that model's default safety thresholds are apparently looser. On 1900 observations across 11 sessions: 0 compression failures, 0 circuit-breaker events.

But:

• it's a preview model (no SLA, behavior may change without notice),
• it doesn't help users on regions/projects without preview access,
• it doesn't fix the underlying observability gap (silent filter → empty content is still reported as unexpected response, not safety_block).

Proposed enhancement

Expose an env var that propagates safetySettings through to the Gemini call.

Gemini's OpenAI-compat endpoint at https://generativelanguage.googleapis.com/v1beta/openai/chat/completions accepts safety overrides via extra_body:

{
  "model": "gemini-2.5-flash",
  "messages": [...],
  "extra_body": {
    "safetySettings": [
      { "category": "HARM_CATEGORY_HARASSMENT",        "threshold": "BLOCK_NONE" },
      { "category": "HARM_CATEGORY_HATE_SPEECH",       "threshold": "BLOCK_NONE" },
      { "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT", "threshold": "BLOCK_NONE" },
      { "category": "HARM_CATEGORY_DANGEROUS_CONTENT", "threshold": "BLOCK_NONE" }
    ]
  }
}

src/providers/_openai-shared.ts (where the request body is built) is the single touchpoint. The patch is small — read 1–N env vars, conditionally append extra_body when the provider is "gemini".

API surface options

Pick whichever matches the repo's taste:

1. Single coarse knob (simplest):

   GEMINI_SAFETY_THRESHOLD=BLOCK_NONE   # or BLOCK_ONLY_HIGH | BLOCK_MEDIUM_AND_ABOVE | BLOCK_LOW_AND_ABOVE | OFF

   Applied uniformly to all four HARM_CATEGORY_*. Unset = current behavior (Gemini's defaults).

2. Per-category overrides (more flexible):

   GEMINI_SAFETY_HARASSMENT=BLOCK_ONLY_HIGH
   GEMINI_SAFETY_HATE_SPEECH=BLOCK_NONE
   GEMINI_SAFETY_SEXUAL=BLOCK_NONE
   GEMINI_SAFETY_DANGEROUS=BLOCK_ONLY_HIGH

3. JSON pass-through (escape hatch):

   GEMINI_SAFETY_SETTINGS_JSON='[{"category":"HARM_CATEGORY_HARASSMENT","threshold":"BLOCK_NONE"}]'

My vote: 1 + 3 — coarse-knob covers ~95% of cases; JSON pass-through for the long tail (custom Vertex AI proxies, future categories the env vars don't cover yet).

Stretch: distinguish safety_block from unexpected response

Even with safety configured down, occasional blocks will happen — and users running with defaults will hit this too. Today the log just says unexpected response, which:

• masks the root cause (anyone debugging will think the API itself is broken),
• amplifies the cascade (no way to know whether to retry, change input, change model, or stop).

A one-line classifier in the provider could turn that into:

error Compression failed {"obsId":"obs_…","error":"safety_block","detail":"empty message.content with finish_reason=stop"}

That's enough for log-grep diagnosis without false certainty about which category triggered (Gemini's OpenAI-compat path doesn't surface that, only the native API does).

Optional follow-up: classify safety_block as non-circuit-breakable, so a single tripped piece of content doesn't shut down compression for unrelated observations. This dovetails with #845: when something other than a real outage fails, route to the synthetic path without poisoning the breaker.

Happy to PR

If this direction looks right, I can put together a PR matching the existing provider style:

• patch contained to src/providers/_openai-shared.ts (+ a couple of env[…] reads in src/config.ts),
• no new deps,
• behavior unchanged when the new env vars are unset.

Let me know which API shape you prefer (1, 2, 3, or 1+3) and whether the safety_block classifier is in-scope for the same PR or a follow-up.

Versions / context

• agentmemory: 0.9.26
• iii: 0.11.2
• Deploy: self-built Docker image (node:22-slim + iii binary + @agentmemory/agentmemory npm) on Kubernetes
• Trigger model: gemini-2.5-flash (default), via Google's OpenAI-compat endpoint
• Workaround model: gemini-3-flash-preview (silently working on the same corpus)
• Sample size: 1903 observations across 11 historical Claude Code sessions, mixed English / Italian