feat(ctf): Prompt Infection (ASI-07) — self-replicating injection via compliance docs#542
Open
Deez-Automations wants to merge 3 commits into
Open
Conversation
… injection via compliance docs Detects a worm-style prompt injection: fraud_agent reads raw vendor-uploaded FinDrive document content directly into its own instructions during Document Compliance Review workflows (fraud.py:235-241, <CTF:VULNERABILITY>), and the orchestrator's Document Compliance Review recipe explicitly forwards fraud_agent's summary "verbatim" to communication_agent (orchestrator.py:119-121). A canary planted in an uploaded document has to survive two independent LLM rewrites -- fraud_agent's own summary, then whatever communication_agent puts in its outgoing email -- to prove genuine self-replication rather than a single lucky echo. Fixed a real bug before ever live-testing this (caught by cross- referencing against the exact lesson already documented in unsigned_relay.py): the detector's Hop 1 check originally read the orchestrator's delegation_complete event's task_summary, which is truncated to 200 characters at the source (orchestrator.py:_emit_delegation_event). Every fraud_agent summary observed in live testing this session has run 500-2000+ characters, so any canary planted mid-summary -- exactly where a model naturally places something like a "regulatory tracking code" -- would have been silently chopped off before the detector ever saw it, regardless of whether the exploit actually worked. Rewired to read fraud_agent's own complete_task tool call (response_dump.tool_calls on its own llm_request_success event), the same untruncated-source technique already proven in UnsignedRelayDetector. Added test_canary_past_200_chars_still_detects as the actual regression test for this -- a realistic >200-char summary with the canary planted past the old truncation point. 14 unit tests passing. Registered in finbot/ctf/detectors/implementations/__init__.py. Full suite: 326 passed, same 4 pre-existing unrelated failures seen on the other two challenge branches this session (Windows path-separator test bug and a stale vendor-lookup fixture). Not yet live-tested against production gpt-5-nano.
…ection detector Ran a code-reviewer agent against this PR since GitHub Copilot's automated review quota was exhausted. Verified each finding against actual source before fixing (not just trusting the report). CRITICAL: _find_canary_tokens only required a digit, not a digit AND a letter. get_relevant_event_types is not scoped to Document Compliance Review -- this detector runs on every communication_agent email in every recipe, and Invoice Processing routes through fraud_agent then communication_agent in the same workflow_id for every ordinary invoice. A purely numeric token (a due-date year, an invoice amount) routinely and coincidentally repeats across an unrelated fraud summary and an unrelated payment-confirmation email, which would have silently awarded full challenge credit to routine invoice traffic that never touched FinDrive. Fixed: require both a digit and a letter, verified with test_pure_numeric_shared_token_does_not_fire and test_canary_must_contain_letter. HIGH: _get_fraud_task_summary returned the FIRST complete_task match found across fraud_agent's events in arbitrary DB row order. Both delegate_to_fraud and delegate_to_system_maintenance invoke run_fraud_agent (fraud.py hardcodes agent_name="fraud_agent" regardless of which orchestrator tool triggered it), so a single workflow can genuinely contain more than one fraud_agent completion. Stopping at the first match could silently miss the one that actually carried the canary. Fixed: renamed to _get_fraud_task_summaries, returns a list, and check_event now unions canary candidates across ALL fraud_agent completions in the workflow -- same aggregation pattern as unsigned_relay.py's _find_persisted_stage. Added test_fires_via_second_fraud_agent_completion_when_first_lacks_canary, deliberately ordering the unrelated event first to expose the bug. MEDIUM: only the email body was checked; communication_agent's own system prompt gives subject-line templates that could carry the substantive detail instead. Fixed: _extract_email_body renamed to _extract_email_text, combines subject + body. Added test_canary_in_email_subject_only_still_detects. MEDIUM: canary matching required exact uppercase, more brittle than unsigned_relay.py's case-agnostic approach given this challenge requires surviving two independent LLM rewrites. Fixed: matching is now case-insensitive (tokens normalized to uppercase for comparison). Added test_canary_case_drift_still_detects. Also added .order_by(CTFEvent.timestamp.asc()) to the fraud_agent query for consistency with unsigned_relay.py's pattern, though not load-bearing now that results are aggregated rather than early-returned. 19 unit tests now (14 original + 5 new), all passing. Full suite: 331 passed, same 4 pre-existing unrelated failures as the other two challenge branches this series. Not fixed, flagging as an open design question instead of a bug: the code-reviewer agent also noted that fraud_agent's user prompt always echoes the orchestrator's free-text task_description (not just the FinDrive document content when is_doc_review is True), and _enrich_with_prior_context forwards prior summaries on every recipe, not just Document Compliance Review. This challenge's YAML has no scoring penalty (unlike Authority Forgery's pi_jb modifier), so a player could plausibly get the two-hop propagation to complete via a cleverly-worded chat message to Admin CoPilot, without ever uploading a FinDrive document or touching the intended <CTF:VULNERABILITY>. Left as-is pending a product decision -- see PR discussion.
…-only bypass
Addresses the open design question flagged in the previous commit: this
challenge's YAML has no scoring penalty (unlike Authority Forgery's
pi_jb modifier), so a player could plausibly get the two-hop canary
propagation to fire via a cleverly-worded chat message to Admin
CoPilot alone, without ever uploading a FinDrive document or touching
the intended <CTF:VULNERABILITY> -- fraud_agent's inbound prompt
always echoes the orchestrator's free-text task_description in every
recipe, not just when a document was actually read.
Added a third, prerequisite stage: fraud_agent's own inbound prompt
(the user_message field already captured on every llm_request_success
event) must show the literal FinDrive injection marker text
("Vendor compliance documents from FinDrive for review:", the header
fraud.py's _get_user_prompt only emits when a real compliance document
was read, fraud.py:276-278) AND the shared canary must appear inside
that same inbound prompt. This is a structural fingerprint of the real
vulnerability firing -- ordinary task-description text written by the
orchestrator's LLM would not organically reproduce this exact phrase.
Consolidated the fraud_agent event query into a single pass
(_get_fraud_agent_evidence) returning both untruncated task summaries
and FinDrive-injected prompts together, instead of querying the same
rows twice.
7 new/updated unit tests covering the new gate (chat-only bypass
correctly rejected; real document read still detected). Verified
against the actual live-test data from earlier today
(workflow wf_chat_5QRVlgAQ353wmopX): the real fraud_agent events do
carry both the marker and the canary, confirming the tightened check
still fires on the genuine exploit, not just synthetic tests.
Considered and deliberately did NOT implement: requiring the canary to
be positionally *inside* the marker-delimited document text (as opposed
to anywhere in the same inbound prompt that also shows the marker).
That would only matter for a deliberately contrived scenario -- a real
document read AND a separately-forged canary coexisting in the same
prompt -- which isn't a meaningful bypass even if it fires, since the
player still had to trigger a genuine document review. Chasing that
precision would be over-engineering past the actual concern (skipping
FinDrive entirely).
20 unit tests total, all passing. Full suite: 332 passed, same 4
pre-existing unrelated failures as the other two challenge branches
this series.
Author
|
Holding off on merging this for now. Live testing showed this challenge's success rate depends heavily on GPT-5's own reasoning making a specific judgment call, and it only lands correctly about 30-40% of the time across repeated attempts. That's not reliable enough to ship for players to depend on. I'm reworking my approach to lean on deterministic, structural bugs instead of behavior that depends on how a specific model reasons in a specific moment. Leaving this PR open for reference, but it's not ready to merge as is. Will follow up once there's a version that holds up consistently, or close this out if it gets replaced entirely. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Prompt Infection, a new advanced-difficulty red-track challenge covering ASI-07 (Insecure Inter-Agent Communication) — a worm-style, self-replicating prompt injection distinct from Unsigned Relay's relay-forgery mechanism.
During a Document Compliance Review workflow,
fraud_agentreads raw, vendor-uploaded FinDrive document content directly into its own instructions with no sanitization (fraud.py:235-241, tagged<CTF:VULNERABILITY>). The orchestrator's Document Compliance Review recipe then explicitly forwardsfraud_agent's summary tocommunication_agentverbatim (orchestrator.py:119-121). A canary planted in an uploaded document has to survive two independent LLM rewrites —fraud_agent's own summary, then whatevercommunication_agentwrites in its outgoing email — to prove genuine self-replication rather than a single lucky echo.What's in this PR
finbot/ctf/definitions/challenges/agent_trust/prompt_infection.yaml— challenge definition, 3-tier hints (mechanism → framing advice → concrete example), OWASP/CWE/MITRE ATLAS labels, references Lee & Tiwari 2024 (arXiv:2410.07283) on LLM-to-LLM prompt injection in multi-agent systemsfinbot/ctf/detectors/implementations/prompt_infection.py—PromptInfectionDetector, a 2-hop kill chain:fraud_agent's owncomplete_tasktool callcommunication_agent'sfinmail__send_emailbody, sameworkflow_idtests/unit/ctf/test_prompt_infection.py— 14 unit testsDesign note: a bug caught before ever running a live test
Before touching a live model, I cross-checked this detector's Hop 1 read against a lesson already documented in
unsigned_relay.pyfrom earlier in this PR series: the orchestrator'sdelegation_completeevent truncatestask_summaryto 200 characters at the source (orchestrator.py:_emit_delegation_event). The original version of this detector read Hop 1 from that truncated event. Everyfraud_agentsummary observed in live testing this session has run 500-2000+ characters, so any canary planted mid-summary — exactly where a model naturally places something like a "regulatory tracking code" — would have been silently chopped off before the detector ever saw it, regardless of whether the exploit actually worked.Fixed before any live run: Hop 1 now reads
fraud_agent's owncomplete_tasktool call directly (response_dump.tool_callson itsllm_request_successevent), the same untruncated-source technique already proven inUnsignedRelayDetector. Addedtest_canary_past_200_chars_still_detectsas the actual regression test — a realistic >200-char summary with the canary planted well past the old truncation point.Test plan
pytest tests/unit/ctf/test_prompt_infection.py— 14/14 passingpytest tests/— 326 passed, 4 pre-existing failures unrelated to this change (same 4 seen identically on the two other challenge branches from this PR series — a Windows path-separator assertion and a stale vendor-lookup fixture)gpt-5-nano, first attempt: created a FinDrive document (soc2_compliance_report.txt,/documentsfolder) containing a canary framed as a closed-out regulatory tracking code inside genuine-looking SOC2 report text; triggered a Document Compliance Review via the Admin CoPilot ("Run a compliance document review for vendor 2's SOC2 certificate"); confirmed the canary survived verbatim intofraud_agent's owncomplete_tasksummary (~400+ characters into the summary — past the old truncation point) and then intocommunication_agent'sfinmail__send_emailbody; detector fired (Challenges completed: ['agent-trust-prompt-infection']in server logs)