Skip to content

prospect diff: compare nodes across versions, runs, and points in time #53

Description

@elecnix

Builds on #50 (as-of reads). Part of the point-in-time set with #51 and the retraction issue.

Why a diff here is stronger than a text diff

Content-addressing makes a difference attributable, not merely visible:

observation meaning
same input_key, different output_key identical recipe — the model reached a different conclusion
different input_key the recipe changed; compare its components to say which

And the components are already separated: analyzer version, config fingerprint, source-set hash. So a diff can state precisely why two nodes differ — the author bumped the analyzer, the user changed config or model, or the inputs moved. A text diff cannot distinguish those, and that distinction is usually the whole question.

Proposed change

prospect diff --unit <analyzer> <source_set_hash>   # across the revises chain
prospect diff --runs <A> <B>                        # what two runs concluded differently
prospect diff --as-of <T1> <T2>                     # added / removed / changed by analyzer

Node content is canonical JSON, so comparison should be structural (changed fields) rather than line-based, with a summary line per node and details on request.

Motivating uses, all encountered in practice

  • "How did our conclusion about this turn change when the prompt improved?" The revises chain already links versions of one logical unit; nothing surfaces the content delta. Answering it currently means reading content_json by hand.
  • Model and prompt bake-offs. Because the resolved model is part of the config fingerprint, different models' verdicts on the same term already coexist as separate nodes. Comparing them is the natural read — and it was recently done by hand-writing a throwaway SQL harness, when --runs A B would have been a first-class command. The same applies to comparing prompt versions: v1.1 vs v1.2 verdicts sit side by side already.
  • Regression triage. When proposal quality moves, --as-of localises when, and --unit shows what actually changed in the conclusion.

Notes

  • --runs needs run ids to be discoverable; prospect analyzers or a prospect runs listing should show recent runs with their mode, model spec, and node counts.
  • For analyzers with many nodes per session, default to a summary (counts of added / removed / changed per analyzer) and require a selector for full content diffs.
  • Diffing should read through the same live/as-of filters as prospect: --as-of reads for querying the graph at a point in time #50 and the retraction issue, so a retracted node appears as removed at T, not as though it never existed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions