feat(analysis): add deterministic preflight planning and local run reporting

[KumamuKuma]

Summary

This PR adds deterministic preflight planning and persistent run telemetry to /understand. Users can review the selected workload before LLM-backed file analysis begins, then inspect a durable report covering stage progress, batch outcomes, retries, failures, and actual processed scope.

Large repositories can otherwise start unexpectedly expensive analysis runs with little visibility into the selected scope or batch workload. These contracts also provide a shared foundation for future budget enforcement, checkpointing, and dashboard progress views.

Linked issue(s)

Refs #284 and #413. Related to Discussion #118.

What changed

Deterministic preflight planning

After scanning and semantic batching, but before any file-analyzer is dispatched, /understand now writes analysis-plan.json with:

• repository-wide and selected file counts
• current lines of code, source bytes, languages, and categories
• content-sensitive selected-source and plan digests
• batch count, distribution, concurrency, and minimum execution waves
• serialized deterministic file-analyzer context bytes
• a low-confidence Phase 2 input-token range
• large-context and input-quality risks
• explainable scope-reduction suggestions

The selected-source digest is calculated from current file contents. Same-size source edits and changes between scanning and planning therefore produce a different inputDigest and planId.

Preflight decision gate

Interactive runs require one of three decisions before file analysis begins:

• Continue runs the displayed workload.
• Adjust scope closes the current preflight run and requires a fresh scan, batching pass, and plan.
• Cancel stops before any file-analysis dispatch.

Non-interactive hosts cancel rather than implicitly accepting an unreviewed expensive run. A deterministic-only option is not exposed because the production pipeline does not currently implement a complete deterministic-only analysis mode.

Persistent run telemetry

Each run writes a versioned run-report.json covering scan, batching, analysis, merge, assembled-graph review, architecture, tour, final review, and save stages.

Stage and batch records retain status, timestamps, duration, attempts, retries, and bounded failure history. The report distinguishes running, interrupted, cancelled, degraded, failed, and successful outcomes.

A new run archives its predecessor and marks an unfinished predecessor as interrupted. Report updates use an exclusive lock and temporary-file plus backup-and-restore writes to preserve the previous valid report if delivery fails.

Estimate boundaries

The preflight token range covers only known Phase 2 input derived from selected source bytes and deterministic batch context. It does not include model output, reasoning, tool protocol overhead, retries, later agents, cache behavior, or provider-specific tokenization.

Wall-time and USD estimates remain unavailable until the host provides calibration data or authoritative model telemetry. Estimated tokens are stored separately from measured usage so consumers cannot mistake an estimate for actual consumption.

Workflow and benchmark compatibility

• Full, incremental, and review-only workflows are instrumented.
• Project scans always exclude .ua/ and legacy .understand-anything/ analysis artifacts.
• The preflight planner reuses only the benchmark's pure serialized-payload-byte calculation.
• The deterministic benchmark report schema, fields, and existing semantics are unchanged.
• On Windows, root Vitest files run serially and the two shebang-bearing analysis CLIs keep LF endings; non-Windows test files stay parallel.

Generated artifacts

The active analysis data directory now retains:

• analysis-plan.json
• run-report.json

Both artifacts use versioned JSON Schema Draft 2020-12 contracts. Existing projects using .understand-anything/ keep that directory, while new projects use .ua/.

How I tested this

• pnpm lint
• pnpm --filter @understand-anything/core test — 970 passed
• pnpm test — 513 passed, 15 skipped
• Manual LLM-backed smoke test — not run to avoid dispatching a paid model workload; production CLI contracts and full/incremental/review orchestration are covered by automated tests

Additional validation:

• core build passed
• plugin skill build passed
• JSON Schema validation and failure-path tests passed
• diff and whitespace validation passed
• GitHub Actions CI run #626 passed on ubuntu-latest and windows-latest

Versioning

• Version bumped from 2.9.4 to 2.9.5 in all six manifests listed in CLAUDE.md
• N/A — internal/docs-only change

Follow-up opportunities

The new plan and report contracts provide a foundation for future work without including those features in this PR:

• complete deterministic-only analysis
• provider and model-specific cost calibration
• hard token, batch, or time budgets
• checkpoint and resume execution
• dashboard progress and historical run views

[KumamuKuma]

Hi @Lum1104, could you please review this PR? It adds deterministic preflight planning and run telemetry, and all CI checks are passing. Thanks!

[thejesh23]

This might gets tricky if someone don't want to emit any of their data especially any software company so we need to have to flag to turn it off.
For example in strix repo , Anything is sent but only send stats like token usage (all of this is explained in telemetry.md) — but can also turn this off with STRIX_TELEMTRY 0 flag
https://github.com/usestrix/strix/blob/main/strix/telemetry/README.md

[KumamuKuma]

Thanks for raising this, @thejesh23 — I agree that the word “telemetry” commonly implies remote data collection.

I checked the implementation end to end. In this PR, it is local run reporting only:

• analysis-plan.json and run-report.json are written only to the project’s local .ua/ directory, or the existing legacy .understand-anything/ directory.
• The planning and run-report helpers make no network requests and do not transmit project, usage, or report data to the Understand Anything maintainers or any analytics service.
• The schemaUrl values stored in the files are identifiers only; the helpers do not fetch them.

I have renamed the PR to use “local run reporting” and documented the privacy and data-transmission boundary in commit e9455c9.

The existing LLM-backed analysis is governed separately by the user’s host and model-provider configuration. Since this PR introduces no remote telemetry or analytics collection, I have not added an opt-out flag. If you also mean the ability to disable creation of the local report files, we can discuss that as a separate requirement.

[Lum1104]

I don't quiet get this, can you provide some detail explaination?

[KumamuKuma]

@Lum1104 Sure — here is a concrete walkthrough of what changes during a normal /understand run.

Before this PR

After scanning and batching the repository, /understand proceeds directly into the expensive file-analysis phase.

For a large repository, the user cannot easily see in advance:

• how many files were selected;
• how many analysis batches will run;
• how many concurrency waves are required;
• approximately how much known Phase 2 input will be processed; or
• whether narrowing the scope would make the run more manageable.

The user also does not receive a durable report showing which stages and batches succeeded, failed, or were retried.

After this PR

The workflow becomes:

scan → batching → preflight plan → user decision → file analysis → graph generation
                         ↓
          Continue / Adjust scope / Cancel

1. Preflight plan

After scanning and batching—but before dispatching any file-analyzer—the workflow writes .ua/analysis-plan.json.

For example, it might report:

Repository files: 5,000
Selected files: 800
Analysis batches: 20
Concurrency: 5
Minimum execution waves: 4
Known Phase 2 input estimate: 180k–360k tokens

It also records source bytes, lines, languages, file categories, batch-size distribution, risks, and possible scope-reduction suggestions.

The plan includes a digest derived from the current selected source. If the source changes, the digest and planId change, so the reviewed plan remains tied to the workload it describes.

2. User decision

The user must then explicitly choose:

• Continue — run the displayed workload.
• Adjust scope — stop the current plan and create a new scan and plan for a narrower scope.
• Cancel — stop before file-analysis dispatch.

A non-interactive host cancels instead of automatically accepting an unreviewed expensive run.

3. Local run report

If the user continues, .ua/run-report.json is updated throughout the workflow.

It records the status and duration of stages such as scanning, batching, analysis, merging, architecture generation, review, and saving. It also records per-batch attempts, retries, failures, and the number of files actually processed.

For example, if batch 7 fails once and succeeds on retry, that history remains visible in the report. If some batches still fail, the final run can be marked degraded instead of incorrectly appearing fully successful.

This report is local only; it is not uploaded or sent to an analytics service.

Important boundaries

This PR does not yet implement:

• hard token, time, or USD budgets;
• checkpoint and resume;
• complete deterministic-only analysis; or
• remote telemetry/analytics.

The token range covers only the known Phase 2 input. It does not claim to estimate model output, reasoning, retries, later agents, cache behavior, wall time, or total USD cost.

So the main purpose of this PR is not to reduce cost automatically. It makes the selected workload reviewable before file analysis begins and provides a durable, machine-readable run report that future budget enforcement, resume support, and dashboard progress views can build on.

The detailed contract is documented here:

https://github.com/Egonex-AI/Understand-Anything/blob/feat/analysis-preflight-telemetry/docs/analysis/preflight-and-run-report.md

[Lum1104]

The main concern is the token estimation. Agents working on graph generation is going to read relevant codebase, current PR does not count these token (while they are the largest token machine).

[thejesh23]

@KumamuKuma Thanks for clarifying my concern on telemetry I'm good as long we don't do any remote data collection. As I'm travelling couldn't review the latest code but I agree with @Lum1104 on token estimation as I myself burned Approx $138 on my first run of /ua on a monorepo.

[ZebangCheng]

I tested the preflight token range against a real Phase 2 run on Emotion-LLaMA to see whether it gives users a reliable cost signal. The estimate topped out at 296K input tokens, but actual uncached input was 637K and cached input reached 7.47M, so the current range is counting a different thing from real usage; I don’t think we should merge this as a token consumption estimate, and I’m closing this PR for now.