Feature/neurocore integration

.gitignore

@@ -71,7 +71,6 @@ config.local.json
 
 # Temporary files
 *.tmp
-<<<<<<< HEAD
 *.temp
 
 # Web development
@@ -81,11 +80,25 @@ out/
 .env*.local
 .npm/
 
-# Build artifacts
+# Build artifacts & Compiled Binaries
 dist/
 build/
 *.egg-info/
-
-=======
-*.temp 
->>>>>>> main
+recon/target/
+network/raw_socket
+network/packet_crafter
+network/tcp_probe
+exploits/shellcode/*.o
+exploits/shellcode/*.bin
+
+# Local Models & Data
+models/
+session/logs/
+reports/
+
+# Tests
+tests/
+
+# Temporary / Extraneous directories
+sim/
+system-prompts-and-models-of-ai-tools/

.planning/PROJECT.md

@@ -0,0 +1,39 @@
+# NeuroRift: llama.cpp Migration Milestone
+
+## Overview
+NeuroRift is an orchestrated multi-agent intelligence system for security research. This milestone focuses on migrating from Ollama to a local `llama.cpp` HTTP server to better leverage specialized hardware (RTX 2050 4GB VRAM) and achieve higher operational control.
+
+## Core Value
+High-performance, local-first AI orchestration for security tools with reduced dependency overhead.
+
+## Requirements
+
+### Validated
+- ✓ Multi-agent orchestration (Planner, Operator, Analyst, Scribe)
+- ✓ Security tool integration (Nmap, ProjectDiscovery)
+- ✓ Web Dashboard (Next.js) & CLI
+- ✓ Persistent session and artifact management
+- ✓ Dockerized microservice architecture
+
+### Active
+- [ ] Replace Ollama with `llama.cpp` HTTP server (localhost:8080)
+- [ ] Implement `llama_client.py` OpenAI-compatible wrapper
+- [ ] Automate model download (Hermes-2-Pro-Mistral-7B-GGUF)
+- [ ] Create `start_llama.sh` with optimized GPU offloading (n_gpu_layers 20)
+- [ ] Update all modules and README for the new stack
+
+### Out of Scope
+- Migrating the web UI to a different framework
+- Adding new security tools in this phase
+- Changing the Rust execution engine logic
+
+## Key Decisions
+
+| Decision | Rationale | Outcome |
+|----------|-----------|---------|
+| llama.cpp vs Ollama | Better VRAM management for RTX 2050 and finer control over n_gpu_layers | Active |
+| OpenAI-compatible bridge | Enables easier integration with existing Langchain/Requests logic | Active |
+| Hermes-2-Pro-Mistral-7B | Best-in-class tool calling capabilities for a 7B model | Active |
+
+---
+*Last updated: 2026-03-22 after initialization*

.planning/REQUIREMENTS.md

@@ -0,0 +1,28 @@
+# Requirements: llama.cpp Migration
+
+## 1. Core Integration
+- [ ] Install `llama-cpp-python[server]` on host system.
+- [ ] Implement `ai_wrapper/llama_client.py`:
+  - OpenAI-compatible chat completions interface.
+  - Support for `tools` (function calling) via JSON schema.
+  - Graceful context overflow handling (truncation or error reporting).
+  - Default parameters: temp 0.1, max_tokens 4096.
+
+## 2. Model Management
+- [ ] Create `scripts/download_model.sh`:
+  - Use `huggingface-cli` or `wget` to fetch `NousResearch/Hermes-2-Pro-Mistral-7B-GGUF`.
+  - Target: `models/hermes-2-pro-mistral-7b.Q4_K_M.gguf`.
+
+## 3. Server Orchestration
+- [ ] Create `scripts/start_llama.sh`:
+  - Launch `python -m llama_cpp.server`.
+  - Config: `n_gpu_layers 20`, `n_ctx 4096`, port 8080.
+
+## 4. Codebase Migration
+- [ ] Audit `modules/` for any hardcoded Ollama endpoints or unique Ollama-client logic.
+- [ ] Refactor `ai_wrapper/` to export the new `LlamaClient` as the primary interface.
+- [ ] Update `requirements.txt` and `setup.py` (remove `ollama`, add `llama-cpp-python`).
+
+## 5. Documentation
+- [ ] Update `README.md` with new prerequisites and startup sequence.
+- [ ] Update `DOCKER.md` (if relevant) to reflect local server dependency.

.planning/ROADMAP.md

@@ -0,0 +1,28 @@
+# Roadmap: NeuroRift llama.cpp Migration
+
+## Milestone 1: Engine Migration
+Migrating the core AI inference engine from Ollama to a local llama.cpp server.
+
+### Phase 1: Environment & Infrastructure [QUICK]
+- **Goal:** Set up the model and server scripts.
+- **Tasks:**
+  - Create `download_model.sh` and fetch weights.
+  - Create `start_llama.sh` and verify server boots on RTX 2050.
+  - Install dependencies.
+
+### Phase 2: Client Implementation [QUICK]
+- **Goal:** Implement the OpenAI-compatible bridge.
+- **Tasks:**
+  - Create `ai_wrapper/llama_client.py`.
+  - Implement tool-calling parser for Hermes-2-Pro.
+  - Unit tests for the new client.
+
+### Phase 3: Total Integration [QUICK]
+- **Goal:** Replace all Ollama call sites.
+- **Tasks:**
+  - Refactor all `modules/` to use `LlamaClient`.
+  - Remove `ollama` package and cleanup legacy configs.
+  - Final end-to-end verification.
+
+---
+*Last updated: 2026-03-22*

.planning/STATE.md

@@ -0,0 +1,19 @@
+# Project State
+
+## Project Metadata
+- **Status:** Initialized
+- **Last activity:** 2026-03-22 - Project Initialization
+- **Next milestone:** Milestone 1: Engine Migration
+
+## Quick Tasks Completed
+| # | Description | Date | Commit | Directory |
+|---|-------------|------|--------|-----------|
+
+## Roadmap Progress
+- Milestone 1: Engine Migration (0% complete)
+  - [ ] Phase 1: Environment & Infrastructure [QUICK]
+  - [ ] Phase 2: Client Implementation [QUICK]
+  - [ ] Phase 3: Total Integration [QUICK]
+
+## Blockers/Concerns
+- None

.planning/codebase/ARCHITECTURE.md

@@ -0,0 +1,26 @@
+# Codebase Architecture
+
+## System Design Pattern
+NeuroRift operates on a **Multi-Agent Orchestration Architecture** with human-in-the-loop (HITL) manual oversight. The system is distributed across three main operational planes:
+
+1. **The Core Orchestrator (Python bridge)**: Manages intelligence routing, LLM inferences via Langchain, state storage, task coordination, and module interaction. 
+2. **The Execution Engine (Rust openclaw)**: A high-performance WebSockets gateway that connects the orchestrator to containerized sandboxed environments for safe execution of security tools. Handles streaming of terminal outputs.
+3. **The Control Plane (Next.js Dashboard)**: A modern, real-time web UI that serves as the mission control for analysts to monitor agent actions, approve HITL requests, and execute tools manually.
+
+## Data Flow
+- User interactions (Dashboard or CLI) define an initial `Task` or `Mission`.
+- The `NR Planner` generates an execution plan and updates the centralized Task State Memory.
+- The `NR Operator` attempts to fulfill tasks, utilizing tools like Nmap or ProjectDiscovery via the Rust sandboxed environments.
+- High-risk operations request explicit human authorization via the Bridge REST API to the control plane.
+- The `NR Analyst` analyzes task outputs stored as JSON artifacts.
+- The `NR Scribe` aggregates findings into actionable markdown/PDF reports.
+
+## Subsystems
+- **AI/LLM Integration (`ai_wrapper/`)**: Encapsulates LLM usage into modular clients for local (Ollama) and remote models.
+- **Workflow Modules (`modules/`)**: Contain domains of expertise (e.g., `recon/`, `scan/`, `darkweb/`, `exploit/`).
+- **State Checkpoints**: The architecture persists task states every 5 minutes in `/data/neurorift/sessions` to allow pause/resume natively.
+- **Security Sandboxing**: The `sandbox-runner` container isolates potentially dangerous terminal workflows so arbitrary shell outputs remain distinct from the host file system.
+
+## Entry Points
+- `neurorift_main.py` & `neurorift_cli.py`: The Python entrypoints for command-line parsing and mode switching.
+- `web-ui/src/app/page.tsx`: The primary Next.js page that interfaces via Server-Sent Events/WebSockets to the backend.

.planning/codebase/CONCERNS.md

@@ -0,0 +1,13 @@
+# Codebase Concerns & Technical Debt
+
+## Known Issues & Vulnerabilities
+- **Integration Complexity**: The polyglot microservice architecture spanning Python (AI Orchestration/API), Rust (OpenClaw WebSocket Gateway), and TypeScript (Next.js Dashboard) relies heavily on Docker networks operating perfectly. Any network latency or WebSocket disconnection can result in untracked state.
+- **Beta Phase Volatility**: The README mentions "🚧 THIS PROJECT IS CURRENTLY IN ACTIVE DEVELOPMENT (BETA Phase) 🚧". Core features are functional, but edge cases during complex agent interactions may cause hangs or crashes.
+- **LLM Non-Determinism**: Dependency on LLMs means the `NR Planner`'s execution plan might vary unpredictably between runs, making debugging and reproducing end-to-end bugs difficult.
+
+## Technical Debt
+- **End-to-End Testing**: Current pytest files (`test_ai_features.py`, `test_security.py`) are strictly unit or mocked integration tests. There appears to be a lack of comprehensive end-to-end (E2E) testing that spin up the full Docker compose environment to simulate a real user session.
+- **Secret Management**: While `.env` is used, the configuration requires storing API keys in plain-text inside the file or Docker orchestration, lacking an integrated Vault or KMS solution for extremely robust key handling.
+
+## Fragile Areas
+- **Sandbox Runner Validation**: The bridge assumes the sandbox cleanly captures command executions without side effects on the host. Any escape from the `openclaw-sandbox` could be catastrophic, necessitating extremely stringent monitoring.

.planning/codebase/CONVENTIONS.md

@@ -0,0 +1,18 @@
+# Codebase Conventions
+
+## Architectural Patterns
+- **Agent Roles**: The application divides responsibilities into distinct agent roles (Planner, Operator, Analyst, Scribe).
+- **Asynchronous Processing**: Heavy use of `async`/`await` in Python for concurrency, especially beneficial for executing non-blocking network I/O and subprocess execution.
+- **WebSocket Streaming**: OpenClaw (Rust) communicates execution updates back to the orchestrator and web UI continuously.
+
+## Code Style
+- **Python Formatting**: Relies heavily on `black` and `pylint`. Maximum line lengths enforce readability.
+- **Imports**: standard library imports before third-party ones.
+- **Strong Typing**: Python methods leverage type hinting where possible to improve static analysis and document input/output clearly.
+
+## Error Handling
+- Use of custom exception classes or graceful fallbacks when LLM execution fails.
+- In security contexts (such as XML parsing), safe APIs (`defusedxml`) are strictly mandated.
+
+## Secret Management
+- Secrets (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, etc.) are managed exclusively via environment variables (`.env`) and Docker environment config injection.

.planning/codebase/INTEGRATIONS.md

@@ -0,0 +1,23 @@
+# Codebase Integrations
+
+## AI Model Providers
+- **Local AI**: Fully integrated with `Ollama` for running local large language models (e.g., `llama3.2`). Runs as a dedicated container in the Docker compose stack.
+- **Cloud AI**: Supports major cloud providers via Langchain:
+  - OpenAI
+  - Anthropic
+  - Google GenAI 
+
+## External Security Tools
+NeuroRift orchestrates various external security tools for reconnaissance and vulnerability assessment:
+- **ProjectDiscovery Suite**: Integrates with `subfinder`, `nuclei`.
+- **Nmap**: Network mapping and port scanning.
+- **WhatWeb**: Technology profiling.
+- **FFUF**: Web fuzzing (mentioned in Web Control Plane).
+
+## APIs & Data Feeds
+- **Dark Web Intelligence**: Integrates with the Tor network via a SOCKS5 proxy (`PySocks`, defaults to `socks5h://127.0.0.1:9050`). Uses Robin integration for `.onion` service scraping and semantic search.
+- **Search Engines**: Uses `duckduckgo-search` for OSINT querying.
+
+## System Communication
+- **Bridge API**: The Python `neurorift-core` acts as a central HTTP/REST bridge (`http://neurorift-core:8766`).
+- **WebSockets Engine**: The Rust engine (`openclaw`) acts as a WebSocket gateway (`0.0.0.0:18789` / `18765`) to communicate state to the frontend and execute high-performance tasks.

.planning/codebase/STACK.md

@@ -0,0 +1,32 @@
+# Codebase Tech Stack
+
+## Overview
+NeuroRift is a terminal-based multi-agent intelligence system built on a polyglot architecture combining Python, Rust, and TypeScript/Next.js.
+
+## Languages & Runtimes
+- **Python (3.10+)**: Primary language for the bridge server, orchestrator, and AI module integrations (`modules/`, `neurorift_main.py`).
+- **Rust**: High-performance engine for system-level operations and the OpenClaw gateway (`core/`, `Cargo.toml`).
+- **TypeScript & Node.js**: Used for the Next.js web dashboard (`web-ui/`).
+
+## Frameworks & Libraries
+- **Backend (Python)**:
+  - `FastAPI` & `uvicorn` for the bridge server API.
+  - `aiohttp` and `asyncio` for asynchronous execution.
+  - `langchain-*` (Core, OpenAI, Ollama, Anthropic, Google) for AI agent orchestration.
+  - `rich` and `click` for the advanced CLI and terminal UI.
+- **Frontend (Web UI)**:
+  - `Next.js` (React framework).
+  - `Tailwind CSS` for styling.
+- **Testing (Python)**:
+  - `pytest`, `pytest-asyncio`, `pytest-cov`.
+- **Code Quality**:
+  - `black`, `pylint`.
+
+## Infrastructure & Deployment
+- **Docker & Docker Compose**: The entire stack is containerized, defining services like `ollama`, `neurorift-core` (Python), `rust-engine`, `gateway`, `sandbox-runner`, and `web-ui`.
+- **Network**: Uses bridged Docker networking (`neurorift_net`).
+
+## Security Tools Used
+- `python-nmap` for network scanning.
+- `dnspython` for DNS querying.
+- `defusedxml` to prevent XXE attacks.

.planning/codebase/STRUCTURE.md

@@ -0,0 +1,44 @@
+# Codebase Structure
+
+## Root Directory Map
+The repository is split into distinct logical environments:
+
+### Backend (Python Core)
+- `neurorift_main.py`: Primary application entry and orchestrator init.
+- `neurorift_cli.py`: Command Line argument parsing subsystem.
+- `modules/`: Feature-specific modules categorizing the application's domain logic.
+  - `ai/`: Agent specific logic.
+  - `recon/`: Logic for invoking reconnaissance tools.
+  - `scan/`: Core vulnerability scanning logic.
+  - `darkweb/`: Darkweb OSINT / Robin integration.
+  - `exploit/`: Exploit testing modules.
+  - `session/`: Persistent session tracking.
+  - `web/`: HTTP handlers and FastAPI bridging.
+- `ai_wrapper/`: Abstract interfaces for Langchain integration across multiple model providers.
+- `configs/`: Default configuration schemas (`neurorift_config.json`, etc.).
+- `prompts/`: Template libraries for the individual agents (Planner, Operator, Analyst, Scribe).
+- `utils/`: Common helpers (logging, formatting, networking constants).
+
+### Execution Engine (Rust)
+- `core/`: The workspace root housing Rust packages.
+- `neurorift-core/`: The `openclaw` Rust package acting as the high-throughput WebSocket Gateway (`Cargo.toml`, `src/`).
+
+### Frontend (Next.js)
+- `web-ui/`: The primary React web application.
+  - `src/`: Component, page, and hooks directory.
+  - `package.json`: NPM dependencies.
+  - `tailwind.config.ts`: Dashboard styling schema.
+
+### Deployment & CI/CD
+- `docker/`: Contains specialized Dockerfiles for different environments.
+  - `docker/neurorift/Dockerfile`: Base image for Python backend.
+  - `docker/openclaw/Dockerfile`: Base image for Rust engine.
+  - `docker/web-ui/Dockerfile`: Base image for Next.js dashboard.
+- `docker-compose.yml`: Comprehensive local compose file connecting all microservices (DBs, Ollama, Core, Gateway, Sandbox, Web-UI).
+- `.github/`: CI/CD workflows for validation.
+
+## Naming Conventions
+- Python files follow standard `snake_case.py`.
+- Rust files follow `snake_case.rs`.
+- React components typically use `PascalCase.tsx`.
+- Configuration files end in `.json` or `.json5`. Document templates use Markdown (`.md`).

.planning/codebase/TESTING.md

@@ -0,0 +1,16 @@
+# Codebase Testing
+
+## Frameworks
+- The Python backend uses `pytest` as its primary test runner.
+- The `pytest-asyncio` plugin is essential since much of the core agent execution engine relies on asynchronous operations.
+- `pytest-cov` is utilized to track test coverage.
+
+## Structure
+- Tests reside in the top-level `tests/` directory.
+- `test_*.py` files map generally to subsystems:
+  - `test_ai_features.py`: Validates integration of various Langchain orchestrations.
+  - `test_security.py`: Ensures security protocols (e.g., stopping XXE via `defusedxml` checks) are enforced across the codebase.
+  - `test_notifier.py`: Validates state streaming notifications.
+
+## Execution
+- Typical test execution incorporates automated stubbing and mocking. Network calls or LLM API endpoints during unit tests are expected to be patched out to prevent external requests or costs.

.planning/config.json

@@ -0,0 +1,14 @@
+{
+  "mode": "yolo",
+  "granularity": "coarse",
+  "parallelization": true,
+  "commit_docs": true,
+  "model_profile": "balanced",
+  "workflow": {
+    "research": true,
+    "plan_check": true,
+    "verifier": true,
+    "nyquist_validation": true,
+    "auto_advance": true
+  }
+}