init commit for external agent framework+gateway

[zackcxb]

What does this PR do?

This PR adds a trainer-side agent framework and gateway runtime for multi-turn agent-style rollout in uni-agent, as a downstream integration of verl RFC #5790 and the upstream agent framework PR verl#6299.

Specifically, it:

• adds uni_agent.trainer.framework — AgentFramework abstract base, OpenAICompatibleAgentFramework concrete implementation, and AgentFrameworkRolloutAdapter (satisfies the trainer's agent_loop_manager_class extension point; recipes wire it in via YAML with no per-recipe glue),
• adds uni_agent.trainer.gateway — _GatewayActor / GatewayManager / GatewayServingRuntime for OpenAI-compatible session serving, sticky session routing, tool-parser wiring, and multimodal media accumulation; backend routing delegates to LLMServerClient,
• adds a deepeyes gateway recipe under examples/deepeyes/,
• adds CPU tests covering framework contract, gateway actor / manager behavior, session runtime lifecycle, and multimodal postprocess.

Wave 2 additions (length budget enforcement + OpenAI parity):

• Rollout prompt_length / response_length budget injected into _GatewayActor; continuation turns clamped to remaining budget; budget-exhausted turns materialise a synthetic finish_reason=length response without hitting the backend.
• All error paths return a unified OpenAI-spec error body ({"error": {"message": …, "type": …, "code": …}}); encode/decode failures caught and surfaced as 400.
• Per-request chat_template_kwargs forwarded to apply_chat_template; reasoning_content preserved through _normalize_message and prefix comparison.
• Unsupported OpenAI capabilities (n>1, response_format, tool_choice=required/function) rejected with 400; tool_choice="none" supported (skips tool injection and parser).
• verl submodule bumped to upstream 3c5f6e04 (verl PR #6129: move LLMServerManager out of AgentLoopManager) so reviewers can git submodule update --init without access to a private fork.

Checklist Before Starting

• Search for similar PRs. Paste at least one query link here:
  https://github.com/verl-project/uni-agent/pulls?q=is%3Apr+gateway+framework
• Format the PR title as [{modules}] {type}: {description}
  • Suggested title: [trainer] feat: add agent framework and gateway runtime

Test

PYTHONPATH=$(pwd) pytest tests/uni_agent/trainer/ -q

Result: 64 passed, 6 warnings (framework, gateway, runtime, multimodal postprocess).

Real-rollout evidence from the deepeyes gateway recipe: a 50-step GRPO run on multi-turn multimodal data (Qwen3.5-4B, 7× RTX 3090 train + 1× local judge) produced a real learning curve — critic/rewards/mean moved from ~0.21 at step 1 to ~1.86 by step 50.

API and Usage Example

Public APIs added:

• uni_agent.trainer.framework — AgentFramework, OpenAICompatibleAgentFramework, AgentFrameworkRolloutAdapter, build_agent_framework
• uni_agent.trainer.gateway — GatewayServingRuntime, GatewayManager, GatewayActor

Minimum viable wiring via YAML config:

actor_rollout_ref:
  rollout:
    agent:
      agent_loop_manager_class: uni_agent.trainer.framework.entry.AgentFrameworkRolloutAdapter
    custom:
      agent_framework:
        framework_class_fqn: uni_agent.trainer.framework.framework.OpenAICompatibleAgentFramework
        gateway_count: 1

The adapter calls build_agent_framework() which wires GatewayServingRuntime and the framework subclass from config. The agent runner only needs the gateway base URL:

async def agent_runner(*, raw_prompt, session_runtime, sample_index, **_):
    await run_external_agent(
        base_url=f"http://127.0.0.1:{session_runtime.port}",
        raw_prompt=raw_prompt,
    )

generate_sequences() writes finalized trajectories directly to TransferQueue with key "{uid}_{session_id}_{index}", matching AgentLoopWorkerTQ._agent_loop_postprocess()'s field / tag layout.

Design & Code Changes

High-level changes:

• AgentFramework base class + OpenAICompatibleAgentFramework own session orchestration (create_session → agent_runner → finalize_session), trajectory assembly, multimodal post-processing, reward scoring, and TransferQueue writes. Per-session failures are isolated via asyncio.gather(..., return_exceptions=True) so one bad session does not cancel the rest of the batch.
• _GatewayActor provides OpenAI Chat Completions over sticky sessions with prefix-consistency checks, tool-parser decoding, multimodal media accumulation, and rollout budget enforcement. GatewayManager routes new sessions by least-active count. GatewayServingRuntime owns gateway actor lifecycle and delegates backend routing to LLMServerClient.
• Multimodal trajectory post-process builds trainer-consumable multi_modal_inputs and (4, seq_len) position ids inside the framework, so VLM sessions do not need per-recipe glue.
• AgentFrameworkRolloutAdapter satisfies the trainer's agent_loop_manager_class contract; every recipe wires the same class in YAML — no per-recipe adapter code.

WIP / Follow-up

• GatewayActor default placement strategy (at least one per node) once multi-node validation is in
• Fully Async support

Checklist Before Submitting

• Read the Contribute Guide (if present).
• Add unit tests to cover all new code — 64 CPU tests included, following the *_on_cpu.py naming convention.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files
• Add / update documentation — deferred to a follow-up; inline docstrings ship with this PR.

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive agent framework and gateway system designed to facilitate agentic workflows within a training environment. Key components include a factory for constructing frameworks, an OpenAI-compatible framework implementation that manages sequence generation and trajectory logging, and a gateway system that provides an OpenAI-compatible API for agent interactions. The gateway handles session lifecycle, trajectory buffering, and multimodal data processing. Feedback identifies a critical issue with the incremental token encoding logic in the gateway, which may produce malformed sequences due to assumptions about tokenizer stability and turn separators. Further recommendations include parallelizing reward calculations to improve performance and replacing blocking ray.get calls with asynchronous operations to avoid event loop starvation.

[gemini-code-assist]

The incremental encoding logic is fragile and likely to produce malformed token sequences. Slicing tokens based on the length of a pre-encoded system prompt assumes that the tokenizer is prefix-stable and that the chat template doesn't insert turn separators or special tokens between the system prompt and the first message. Furthermore, concatenating these incremental IDs to the previous turn's response IDs (at line 542) will miss the necessary turn separators (e.g., <|im_end|> and <|im_start|>user) required by most chat templates. It is safer to re-encode the full message history and identify the delta, or simply rely on the backend's prefix caching by sending the full prompt.

[gemini-code-assist]

Using ray.get inside an async context (called via build_agent_framework) will block the event loop, preventing other concurrent tasks from making progress. Since a helper _await_ray_ref is already defined in this file, you should consider moving the gateway startup logic to an async initialization method that can be awaited, rather than performing blocking calls in the constructor.

[wangtiance]

为什么放在trainer目录下？我觉得这是黑盒调用训推通用的流程。我偏向往上提一级，直接放uni_agent/framework和uni_agent/gateway.

[wuxibin89]

Move OpenAICompatibleAgentFramework into a separate file, keep abstract interface only.

[zackcxb]

moved the abstract interface to base.py

[yyDing1]

The current entry point binds a single runner via agent_runner_fqn + agent_runner_kwargs. This works for a single-task recipe like DeepEyes, but it doesn't scale to multi-task rollout.

We may introduce an AgentRunner abstract base with a minimal run() contract:

# uni_agent/trainer/framework/runner.py
class AgentRunner(ABC):
    name: str = ""
    @abstractmethod
    async def run(
        self,
        *,
        raw_prompt: list[dict],
        session: SessionHandle,
        session_runtime: SessionRuntime,
        sample_index: int,
        tools_kwargs: dict[str, Any] | None = None,
    ) -> None:
        ...

Each sample carries the runner name; config mounts a name → runner map.

The config could be in the following format:

# agent_runner.yaml
- name: deepeyes
  _target_: examples.agent_train.deepeyes_gateway.runner.DeepEyesAgentRunner
  max_turns: 5
  tools:
    - name: image_zoom_in_tool
      config_path: examples/agent_train/deepeyes_gateway/configs/image_zoom_in_tool_config.yaml

- name: swe
  _target_: examples.agent_train.swe_gateway.runner.SweAgentRunner
  max_turns: 50
  env:
    deployment:
      type: vefaas
      command: ...
  tools:
    - name: str_replace_editor
    - name: execute_bash

Then the framework resolves the runner per-session by sample["agent_runner_name"], like:

# framework.py:_run_session
runner = self._runners_by_name[sample_fields["agent_runner_name"]]
await runner.run(
    raw_prompt=raw_prompt,
    session=session,
    session_runtime=self.session_runtime,
    sample_index=sample_index,
    tools_kwargs=sample_fields.get("tools_kwargs"),
)

This could be similar to verl's existing agent_loop_config pattern, and we can adopt the same shape here.

[gxlvera]

Hi, I would like to propose using Prefix Trie for multi-trajectory storage for Agentgateway. My RFC is here:#51
This approach could address the following limitations of current implementation:

• Single active branch only: A session keeps one message_history and one active trajectory. When switching sub-agents, picking a resample path, or returning to an older branch, new requests cannot reattach to historical branches. A trie keeps every branch; incoming messages longest-prefix-match against any path and continue from there.
• Repeated encoding of shared prefixes: Message/token prefixes shared across trajectories are re-materialized and re-tokenized on every branch switch. A trie stores checkpoints on shared nodes; later calls clone from the matched node and tokenize incrementally.
• No concurrent inference: One shared state requires a generation lock and serial LLM calls. With a trie, each call owns a cloned branch state; tokenize and commit can interleave—supporting sub-agents, best-of-n, etc.

For detailed explanation, please also refer to this comment: verl-project/verl#6299 (comment)

[sl-1314]

Hi, I noticed that, in the original verl AgentLoopManager/AgentLoopManagerTQ, it spawns num_workers independent AgentLoopWorker(Ray actors), distributing the total train_batch×rollout.n agent loops across these actors for parallelism. However, OpenAICompatibleAgentFramework.generate_sequences currently runs all train_batch×rollout.n sessions in a single asyncio event loop inside the PPOTrainer process. This means CPU-bound operations in one agent loop (e.g. tool execution) will block all other concurrent sessions.

Suggestion: Refer the original verl AgentLoopManager pattern — introduce multiple AgentLoopWorker Ray actors, partition the batch tasks across them.

# refer AgentLoopManager._init_agent_loop_workers()
for i in range(num_workers):
    worker = AgentLoopWorker.options(
        scheduling_strategy=NodeAffinitySchedulingStrategy(node_id=..., soft=True)
    ).remote(config, ...)
    self.workers.append(worker)

async def generate_sequences(self, prompts):
    chunks = prompts.chunk(len(self.workers))
    await asyncio.gather(*[
        w.generate_sequences.remote(chunk)
        for w, chunk in zip(self.workers, chunks)
    ])

This separates two independent concurrency axes: gateway_count for LLM serving throughput, num_workers for agent execution parallelism — consistent with the original verl design.

[wuxibin89]

1. P0: Add doc string for all public class, method, fields and functions.
2. P1: Separate this PR into 3 PRs: gateway, framework, deepeyes examples. We can only review the gateway part before sepration.

[wuxibin89]

Why not just decorate it with @ray.remote?

@ray.remote
class GatewayActor:
    ...

[zackcxb]

Hi, I noticed that, in the original verl AgentLoopManager/AgentLoopManagerTQ, it spawns num_workers independent AgentLoopWorker(Ray actors), distributing the total train_batch×rollout.n agent loops across these actors for parallelism. However, OpenAICompatibleAgentFramework.generate_sequences currently runs all train_batch×rollout.n sessions in a single asyncio event loop inside the PPOTrainer process. This means CPU-bound operations in one agent loop (e.g. tool execution) will block all other concurrent sessions.

Suggestion: Refer the original verl AgentLoopManager pattern — introduce multiple AgentLoopWorker Ray actors, partition the batch tasks across them.

# refer AgentLoopManager._init_agent_loop_workers()
for i in range(num_workers):
    worker = AgentLoopWorker.options(
        scheduling_strategy=NodeAffinitySchedulingStrategy(node_id=..., soft=True)
    ).remote(config, ...)
    self.workers.append(worker)

async def generate_sequences(self, prompts):
    chunks = prompts.chunk(len(self.workers))
    await asyncio.gather(*[
        w.generate_sequences.remote(chunk)
        for w, chunk in zip(self.workers, chunks)
    ])

This separates two independent concurrency axes: gateway_count for LLM serving throughput, num_workers for agent execution parallelism — consistent with the original verl design.

Hi, I noticed that, in the original verl AgentLoopManager/AgentLoopManagerTQ, it spawns num_workers independent AgentLoopWorker(Ray actors), distributing the total train_batch×rollout.n agent loops across these actors for parallelism. However, OpenAICompatibleAgentFramework.generate_sequences currently runs all train_batch×rollout.n sessions in a single asyncio event loop inside the PPOTrainer process. This means CPU-bound operations in one agent loop (e.g. tool execution) will block all other concurrent sessions.

Suggestion: Refer the original verl AgentLoopManager pattern — introduce multiple AgentLoopWorker Ray actors, partition the batch tasks across them.

# refer AgentLoopManager._init_agent_loop_workers()
for i in range(num_workers):
    worker = AgentLoopWorker.options(
        scheduling_strategy=NodeAffinitySchedulingStrategy(node_id=..., soft=True)
    ).remote(config, ...)
    self.workers.append(worker)

async def generate_sequences(self, prompts):
    chunks = prompts.chunk(len(self.workers))
    await asyncio.gather(*[
        w.generate_sequences.remote(chunk)
        for w, chunk in zip(self.workers, chunks)
    ])

This separates two independent concurrency axes: gateway_count for LLM serving throughput, num_workers for agent execution parallelism — consistent with the original verl design.

Thanks for the comment, this is a valid point. The current framework path fans out all
batch_size * rollout.n sessions with asyncio.gather inside the trainer process, and the
existing semaphore only caps coroutine concurrency; it does not isolate CPU-bound tool or
sandbox work. We have also seen this show up with SWE-style runners where sandbox work
can block unrelated sessions.

Since I have split the current PR into three separate PRs (gateway, framework, and examples), I will treat this as a framework-layer follow-up and address it in the next PR instead of this one.
For the framework PR, I plan to keep the reference OpenAICompatibleAgentFramework the lightweight inline implementation but documenting the dispatching mode. Heavy runners such as SWE can provide a framework subclass that overrides the session execution boundary and runs the agent runner
in process/Ray workers.