Skip to content

Harden structured LLM outputs and add preflight CLI input schema vali… - #3

Merged
yycyyv merged 1 commit into
yycyyv:mainfrom
RichardDawald:Check-v2
May 23, 2026
Merged

Harden structured LLM outputs and add preflight CLI input schema vali…#3
yycyyv merged 1 commit into
yycyyv:mainfrom
RichardDawald:Check-v2

Conversation

@RichardDawald

Copy link
Copy Markdown
Contributor

Addresses two reliability gaps:

  1. LLM structured-output robustness at critical nodes (especially traceability), where half-structured JSON could break the chain.
  2. CLI input quality guardrails, by validating --input payloads locally before backend requests.

The goal is fail-soft on model output and fail-fast on bad CLI payloads.

  1. Traceability node: secondary structural repair + minimal field completion

Changes:

  • Updated traceability_check_node to add a second-stage repair path:
    • Run normal agent.run_structured(...).
    • Post-process output with a deterministic repair function to ensure minimum usable structure.
    • If structured generation/validation still fails, return a deterministic minimal ClaimTraceabilityReport instead
      of failing the whole draft chain.

Added helpers:

  • _repair_traceability_min_fields(report_payload, claims_payload)
    • Ensures each claim has a traceability report row.
    • Ensures each report has at least one elements_evidence.
    • Fills missing/empty fields with safe placeholders.
    • Preserves schema compliance via ClaimTraceabilityReport.model_validate(...).
  • _build_minimal_traceability_report(claims_payload, failure_message)
    • Deterministic fallback report when output is unrecoverable.
    • Marks items as Unsupported and requests human review in risk assessment text.
  • _extract_claim_rows_for_traceability(claims_payload)
    • Best-effort extraction of claim numbers/elements for fallback generation.
    • Includes synthetic fallback row if claims are missing.

Behavioral intent:

  • Significantly reduce end-to-end failures caused by half-structured model JSON at traceability stage.
  • Keep workflow progressing with explicit “needs human review” semantics rather than hard-fail.
  1. CLI --input preflight schema validation (before request)

Changes:

  • Added local validators:
    • ensure_nonempty_string_field
    • ensure_array_of_nonempty_strings_field
    • ensure_object_field
  • Added validate_cli_input_schema(workflow, payload) for workflow-specific checks:
    • draft: require idempotency_key and (disclosure_text or disclosure_file_id)
    • oa: require idempotency_key and (oa_text or oa_notice_file_id), optional shape checks for related fields
    • compare: validate optional key types and require meaningful input signal (file ids/paths)
    • polish: require at least one of application_file_id / original_claims / application_specification
  • Hooked validation into execution path:
    • after payload assembly (build_cli_request_payload)
    • before any API call (post_json)
  • Error message prefix:
    • CLI input schema validation failed: ...

Behavioral intent:

  • Early, readable local failures for malformed --input.
  • Fewer invalid backend requests and clearer operator feedback.

Why This Is Safe:

  • Changes are additive and defensive.
  • No API route signature changes.
  • No normal-path degradation: valid model outputs and valid CLI payloads keep existing behavior.
  • Fallback paths are only activated under malformed model output or invalid CLI inputs.

@yycyyv
yycyyv merged commit 8b97580 into yycyyv:main May 23, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants