Rust Port Feature Parity Audit vs Upstream Python SIA (June 2026)

[micahstubbs]

Honest assessment of current gaps between sia_rust and the original hexo-ai/sia Python implementation.

This issue documents areas where the Rust conversion is not finished, not fully functional, or lacks full feature parity with the upstream Python version as of June 6, 2026.

What is at or near full parity (or better)

• Core orchestrator loop, context management, and generation state machine
• Byte-perfect prompt / context.md / trajectory artifact parity (golden-master tests are actually stronger than original)
• Web dashboard (sia web) and run visualization
• CLI interface and runs/ directory layout / artifact compatibility
• Task definition + evaluate.py contract (via Python subprocess bridge)
• Performance (geometric mean ~5.8× faster; up to 25× on prompt building)
• Type safety, compile-time guarantees, and testability

Major Gaps (Not at Feature Parity)

1. Weight Updates / LoRA / Test-Time Training (Highest Impact Gap)

• Completely missing in Rust.
• The original SIA paper and Python implementation support a unified Feedback-Agent loop that performs both harness updates and weight updates (LoRA fine-tuning).
• Without this, sia_rust cannot yet demonstrate the full "SIA-W+H" behavior that is the core contribution of the paper.
• Related: LoRA / Test-Time Training / Weight Update Integration (Native Rust preferred) #19, Real weight updates — current path is a CPU reference toy, not actual model training #91 (blocked on Candle environment in this sandbox).

2. Target Agent Execution

• Still requires a Python runtime + evaluate.py subprocess.
• No native Rust implementation of target agent code generation + execution exists.
• This was a deliberate early architectural decision (Python bridge), but it means full end-to-end Rust execution of the self-improving loop is not achieved.

3. LLM Runners for Meta & Feedback Agents

• Native rig-core implementation exists behind the optional llm Cargo feature.
• Real provider calls are behind #[ignore] tests and require API keys.
• Without the feature flag, meta/feedback agents are non-functional stubs.
• Related: Complete Native End-to-End LLM Client + Rig-Core Integration (blocks live SIA loop) #62, Implement Robust Multi-Provider LLM Client Abstraction with Tool/Function Calling #35, Epic: Native Rust LLM agent runners (replace the SDK integration boundary) #38, Implement core AgentRunner abstraction on rig-core (Phase 1 of LLM client) #50

4. Advanced Sandboxing & Security Model

• Current implementation has a Rust capability allow-list.
• Full production-grade hardening (Landlock + seccomp + WASI for dynamically generated code) is only documented in SECURITY.md, not yet implemented.
• This is a notable gap for a framework that executes self-generated code.
• Related: Security/Sandbox Hardening + Formal Threat Model (Framework track differentiator) #67, Scheduler decision is observational only — make it actually drive the generation loop #90

5. Newer / Hackathon-Driven Features (Not in Original Python)

• MCP tool support (feat(web): run-level scheduler decision timeline in SIA Studio (#85) #104)
• Durable trajectory memory + RAG for Feedback Agent (Claim Nebius Token Factory credits ($50 + $100) for hackathon #105)
• Parallel target-agent evaluation + A/B harness testing (Capability sandbox is defined but never enforced in tools.rs (native tool path is unsandboxed) #89)
• Structured improvement.json output (Native runners never emit telemetry.json — telemetry capture is not wired #88)
• Adaptive scheduler actually driving real weight updates (Wire adaptive scheduler + weight update modules into orchestrator for live closed-loop SIA #84)
• These are valuable extensions but represent work that was never in the upstream Python version.

6. MLE-Bench / External Dataset Bootstrap

• Still largely relies on Python-side scripts and setup.

Overall Verdict

sia_rust has a very strong, clean, and high-performance foundation. The core self-improving harness loop, observability, and testing are solid and in many ways superior to the original.

However, it is not yet at full feature parity on the most novel part of the SIA paper: the unified harness + weight update loop. Completing the weight update path (even via a temporary Python bridge) and hardening the sandbox would bring it much closer to parity while preserving Rust's advantages.

This audit should be kept up to date as work progresses on #62, #84, #19/#91, and the security roadmap.