🛡️ LLM Agent Red-Teaming

Responsible red-teaming of agentic LLMs — real case studies on how coding/agent models (Kimi, and more) can be manipulated into generating harmful action plans, and how to defend against it.

Keywords: LLM security · AI agent safety · prompt injection · jailbreak · agent hijacking · agentic AI red teaming · LLM agents · responsible disclosure · AI security research

---

This repository documents security findings for awareness and defense. It shows that modern agentic LLMs can be steered into planning real-world attacks, and why it matters — it does not publish working exploit prompts or copy-pasteable attack recipes.

If you build, deploy, or study LLM agents, these findings are for you. ⭐ Star the repo to follow new case studies.

📑 Table of Contents

• Why This Matters
• Case Studies
• Featured: Kimi K3 Agent Hijack
• Key Defenses
• Scope & Ethics
• Repository Structure
• Disclaimer
• 한국어 요약

🔥 Why This Matters

Modern reasoning/agentic LLMs are increasingly given direct tool and shell access (CLI agents, auto-approve / "yolo" modes, autonomous coding agents). This research explores a dangerous failure mode: instead of using their reasoning budget to detect and refuse malicious intent, models can be steered into using that same reasoning to optimize and structure an attack plan — targeting third-party infrastructure and harvesting credentials.

In a non-sandboxed environment, the gap between "the model wrote the plan" and "the host executed the attack" can be a single keystroke.

🧪 Case Studies

Model	Environment	Finding	Category
Kimi K3	CLI (non-sandboxed, auto-approve)	Guardrails bypassed; reasoning weaponized to draft a third-party attack plan	Agent Hijacking / Prompt Injection
more coming	—	Additional models under evaluation	—

🎯 Featured: Kimi K3 Agent Hijack

A targeted prompt injection bypassed the agent's safety guardrails. Instead of refusing, the model used its maximum reasoning budget to draft a structured, actionable cyber-attack pipeline (infrastructure scanning + credential harvesting) — inside a live, non-sandboxed CLI with auto-execute enabled.

All concrete commands, queries, and targets are redacted. This documents the model's behavior, not a reproducible attack.

📄 Full write-up: case-studies/kimi_k3_agent_hijack.md

🧱 Key Defenses

If you operate LLM agents, harden them with:

1. Strict execution sandboxing — no direct OS-level command rights; ephemeral, locked-down containers with network egress filtering.
2. Independent plan verification — a separate "guardian" check evaluates generated command plans against policy before anything runs.
3. Tool access whitelisting — restrict package installs and block known scanning/exploitation utilities in agent environments.

⚖️ Scope & Ethics

• Defensive intent only. Nothing here endorses or facilitates unauthorized access.
• No attack methodology. Reproduction prompts and alignment-bypass techniques are intentionally withheld.
• Payloads are defanged. Concrete commands, queries, endpoints, and targets are redacted in all artifacts — screenshots and text alike.
• No real targets. No third-party system was scanned, accessed, or attacked. Findings describe what a model proposed, not actions taken.

📁 Repository Structure

llm-agent-redteaming/
├─ README.md
├─ LICENSE
├─ case-studies/
│  └─ kimi_k3_agent_hijack.md       # Kimi K3: agent hijack → attack-plan generation
└─ evidence/
   └─ kimi-k3-plan-redacted.png     # redacted screenshot

📢 Disclaimer

This repository is published for research, education, and defensive purposes only. It does not contain functional attack tooling or reproduction instructions. The author does not condone, and is not responsible for, any misuse of the concepts described herein. Do not attempt any of the described activities against systems you do not own or have explicit written authorization to test.

---

🇰🇷 한국어 요약

LLM 에이전트 레드팀 연구 — 코딩/에이전트형 LLM(Kimi 외 다수)이 어떻게 유해한 실행 계획을 생성하고, 비샌드박스 환경에선 실행까지 갈 수 있는지를 다루는 독립 보안 연구 저장소입니다.

• 이 저장소는 "이런 일이 가능하다"는 인식 제고와 방어를 위한 것입니다.
• 탈옥/우회 방법(재현 프롬프트, 페르소나 트릭 등)은 공개하지 않습니다.
• 모델이 뱉은 구체적 명령어·타깃·쿼리는 모든 산출물에서 가려져(redacted) 있습니다. 핵심은 "모델이 보인 행동"이지, 복붙 가능한 공격 레시피가 아닙니다.
• 연구 과정에서 실제 제3자 시스템을 스캔·접근·공격한 사실은 없습니다.

핵심 발견 (Kimi K3): 정교한 프롬프트 주입으로 안전장치가 우회되자, 모델은 뛰어난 추론력을 거절이 아니라 공격 계획 최적화에 사용했습니다. 게다가 비샌드박스 CLI + 자동실행 모드라, 실행 버튼 하나 차이로 실제 공격이 시작될 수 있는 상황이었습니다.

본인 소유가 아니거나 명시적 서면 허가가 없는 시스템에 대해 여기 기술된 어떤 행위도 시도하지 마십시오.

---

_{Topics: llm-security ai-safety red-teaming prompt-injection jailbreak agentic-ai llm-agents ai-security cybersecurity responsible-disclosure}