Add periodic A2A worker progress updates

[dimavrem22]

Executive Summary

This PR gives inbound A2A requesters an immediate cadence-aware acknowledgment followed by concise progress updates while Hermes is working.

• Defaults progress updates to every three minutes and states that cadence in the initial receipt.
• Uses a dedicated auxiliary model task with bounded normalized tool identifiers, deterministic fallback, and identifier-echo rejection.
• Preserves authenticated caller input, delivery, cadence, and outcome ordering across retries, follow-ups, process restarts, cancellation, and shutdown.

Description

Every created or message admission first verifies the exact authoritative task, context, active state, and latest caller-role message. The durable payload is rebuilt from the authoritative message parts and task caller metadata, so webhook-supplied caller IDs, organization IDs, handles, and text cannot reach the receipt, registry, or model turn. Inbound A2A admission then sends a nonterminal receipt after durable context binding and before model work begins. While Hermes processes the task, a referenced background task emits short progress replies every 180 seconds by default. The interval is configurable through platforms.inkbox.a2a_progress_interval_seconds or INKBOX_A2A_PROGRESS_INTERVAL_SECONDS; 0 disables periodic updates.

The auxiliary writer receives requester text and up to eight normalized tool names from tool-start hooks. It never receives tool arguments, results, or model reasoning, and generated prose that repeats an identifier is rejected. Candidate updates are persisted before delivery, matched only against exact agent-role history, and retried after five seconds when active delivery fails. Cadence and pending delivery survive restart and caller follow-up.

Explicit completion, input, and failure tools persist a message-owned fence before their side effect and drain an in-flight update before replying. Stop, cancellation, and shutdown close A2A admission before draining active delivery and work. Once closing starts, every A2A event—including cancellation and sent-task updates—receives a retryable 503 before it can mutate state or wake a session. Catch-up enumerates both exact active states, deduplicates task IDs, applies authoritative admission, reuses durable caller input, treats input-required and authentication-required tasks as settled, never mistakes worker history for new work, and rejects replay of the same outcome-fenced caller message before enqueue. Cancellation reconstructs missing message IDs from durable and authoritative state, tombstones only that generation, and permits one distinct authoritative active caller follow-up.

Reason

A requester should be able to distinguish queued, actively running, and finished worker tasks without treating a long silent period as a failed handoff. Neither stale nor spoofed webhook data may choose the caller identity, organization, text, or work generation used by Hermes. Restart recovery must also discover work that has already advanced from submitted to working.

Decisions

• Runtime default: Three minutes balances useful visibility with auxiliary-model cost for ordinary tasks.
• Live override: One minute keeps CI bounded while still proving real timer and nonterminal protocol delivery.
• Admission authenticity: Durable writes, receipts, and model work require exact authoritative task, context, active state, and latest caller message; caller metadata and parts come only from the authoritative task.
• Recovery discovery: Query the exact submitted and working states, deduplicate by task ID, and pass every candidate through authoritative admission.
• Delivery recovery: Candidate text is journaled before sending; only exact agent or ROLE_AGENT history confirms delivery, and uncertain active delivery retries after five seconds.
• Cadence recovery: Scheduling remains anchored to the durable original start time, and pending state transfers to caller follow-ups without regeneration.
• Outcome ordering: Outcome tools persist a caller-message-owned fence and drain active progress before the reply side effect.
• Cancellation recovery: Cancellation blocks only the canceled generation; one distinct authoritative active caller message can resume.
• Shutdown ordering: Shutdown closes all A2A event admission before awaiting admissions, progress delivery, and active worker work.
• Requester behavior: Informational progress remains protocol-visible without waking requester agents.

Testing

• .venv/bin/python -m pytest -q — 602 passed, 25 skipped.
• .venv/bin/python -m pytest -q tests/test_a2a.py — 78 passed.
• python3.11 -m pytest -q tests/test_a2a.py — 78 passed.
• .venv/bin/ruff check . — passed.
• .venv/bin/python -m compileall -q . — passed.
• git diff --check — passed.
• Focused regressions verify working-state recovery after a stale persisted generation, task-ID deduplication across active-state discovery, and retryable post-close rejection of cancellation and sent-task updates without mutation or wake-up.
• Live inbound progress expects prompt acknowledgement, ordered nonterminal updates at the CI 60-second cadence, and a completed answer containing the unique token and 4 + 6 = 10.