Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ _atlas_site/

# local QA/scratch artifacts
.hotato/
hotato-output/
hotato-analyze*.html
hotato-fixplan*.json
hotato-pull-*/
Expand Down
25 changes: 25 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,31 @@ Every entry reports millisecond measurement error and a confusion matrix. See `d
readiness.

### Added
- **`hotato autopsy <recording>`: the zero-config forensics front door.** Drop
one call recording in and get the incident list out -- BARGE-IN, TALK-OVER,
DEAD AIR, LATENCY SPIKE -- each with a timestamp, the measured magnitudes,
and the scanner's plain-English sentence, plus ONE self-contained HTML
report (per-channel waveform with a labeled marker per incident, then a
card per incident) written to a content-addressed path under
`hotato-output/`. A two-channel recording runs the existing deterministic
whole-call scanner unchanged: same file in, byte-identical CLI text and
report out, every run. A mono recording is analyzed best-effort with the
same energy VAD: silence-timing findings (dead air, latency gaps) each
carry a measured confidence with its derivation printed beside it, and the
one-line functional scope states that talk-over/barge-in attribution comes
from a two-channel export -- nothing guessed, no confidence invented, and
the stricter commands (`run`/`scan`/`trust`/contracts) keep refusing mono.
WAV reads natively; mp3/m4a convert through ffmpeg when on PATH (a one-line
actionable message when absent); an unreadable input is refused with the
reason (exit 2). The autopsy id is content-derived (`apx-` + 12 hex of
sha256 of the file bytes) and each incident is addressed `<id>#<rank>` --
printed on the `pin:` line. `est. cost` lines render ONLY with
`--cost-config` (your own per-incident figures; no default dollar amount
exists anywhere). `hotato autopsy` with no arguments prints the quick
start on the bundled examples; `examples/autopsy/` ships three
deterministic rendered example calls (seeded renderer, seed = sha256(id)),
one per pattern: barge-in-say-do, latency-dead-air, talk-over. See
`docs/AUTOPSY.md`.
- **`hotato console` + `/calls`: the daily surface over scored production
calls.** `hotato console --production-db DB` is serve + the production
evidence database + the score-on-arrival worker in one command, one process,
Expand Down
135 changes: 135 additions & 0 deletions docs/AUTOPSY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
# `hotato autopsy <recording>`: one call in, the incidents out

Drop one call recording in with zero config and get the incident list:
barge-in, talk-over, dead air, latency spikes, each with a timestamp and
the measured magnitude, plus one self-contained HTML report.

```bash
hotato autopsy call.wav
```

```
hotato autopsy: call.wav (12.0s, 2 channels, stereo)
[CRITICAL] BARGE-IN t=2.99s overlap=1.96s agent did not go silent within 3.0s
the caller took the floor and the agent kept talking over them for 1.96 s without going quiet within 3.0 s
[WARNING] DEAD AIR t=6.65s trailing silence=1.94s no caller energy within 0.50s
the agent went quiet for 1.94 s with no caller energy nearby
2 incidents: 1 critical, 1 warning
report: hotato-output/autopsy-apx-cc33f46fad58.html
pin: apx-cc33f46fad58 (incidents apx-cc33f46fad58#1..#2)
```

WAV reads natively; mp3/m4a convert through `ffmpeg` when it is on PATH
(when it is not, the one-line message names the install command).
Everything runs offline; no audio leaves the machine. `hotato autopsy`
with no recording prints the quick start on the bundled rendered example.

## Stereo: the deterministic path

A two-channel recording (caller on one channel, agent on the other) runs
the existing whole-call scanner ([`hotato scan`](../src/hotato/scan.py))
unchanged: the same walk, the same measured numbers, byte-for-byte. The
same file produces byte-identical CLI text and a byte-identical report on
every run -- the report is even named by content (see the autopsy id
below), so the path is stable too.

The scanner's candidate kinds map to the incident vocabulary:

| Incident | From | CRITICAL when |
| :-- | :-- | :-- |
| `BARGE-IN` | the caller became active while the agent was talking (`overlap_while_agent_talking`) | the agent kept talking over the caller past the 1.0 s prompt-yield ceiling, or never went quiet in the search window |
| `TALK-OVER` | the agent started a fresh utterance over the caller (`agent_start_during_caller`) | the overlap exceeds the same 1.0 s ceiling |
| `DEAD AIR` | a response gap of 5 s or more (`long_response_gap`), or the agent stopping with no caller energy nearby (`agent_stop_no_caller`) | the gap is 5 s or more |
| `LATENCY SPIKE` | a response gap between 2 s and 5 s (`long_response_gap`) | never (a warning) |
| `ECHO SUSPECTED` | the caller channel tracks the agent's own audio (`echo_correlated_activity`) | never (a caveat) |

Each incident block carries the severity, the timestamp, the measured
magnitudes (`candidate_detail`), and the scanner's plain-English sentence
(`candidate_plain_english`) -- the same measured numbers, restated once.

## Mono: best-effort, confidence-scored

A one-channel (mixed) recording is analyzed best-effort with the same
energy VAD. One mixed channel measures silence timing -- dead air and
latency gaps -- and every mono finding carries a **measured confidence**
with its derivation printed beside it: how far the gap's mean energy sits
below the speech-activity threshold the VAD measured for this recording
(a 20 dB margin or more scores 1.00). Talk-over and barge-in attribution
comes from a two-channel recording, where the caller and the agent are
physically separated; that functional scope is stated once, on one line,
in the output. A mono gap says everything stopped, not who stopped --
nothing is guessed and no confidence is invented.

The stricter commands keep their bar: `run`, `scan`, `trust`, and the
contract path still refuse mono as NOT SCORABLE. Autopsy is discovery;
the CI gate stays deterministic and dual-channel.

An unreadable input -- a text file, a truncated header, a non-audio blob
-- is refused with the reason (exit 2), never scored.

## The report

One self-contained HTML file under `./hotato-output/`, in the
[`hotato report`](REPORTS.md) house style: the per-channel energy
waveform with one labeled marker per incident, then a card per incident
with the same measured numbers the CLI printed. Zero external requests
and zero scripts; the page renders the same bytes for the same recording
on every run.

## The autopsy id

`apx-` + the first 12 hex chars of the sha256 of the input file's bytes:
content-derived, so the same recording gets the same id on any machine,
whatever the file is named (an mp3 hashes its own bytes, so the id is
independent of the local ffmpeg build). Incidents are addressed as
`<autopsy-id>#<rank>` -- the `pin:` line prints both -- and the report is
`hotato-output/autopsy-<id>.html`.

## est. cost, only from your figures

Cost lines render only when you supply your own per-incident figures;
hotato ships no default dollar amount:

```bash
hotato autopsy call.wav --cost-config costs.json
```

```json
{"currency": "USD",
"per_incident": {"dead-air": 3.0, "barge-in": 2.0,
"talk-over": 2.0, "latency-spike": 1.0}}
```

Each priced incident gets an `est. cost` line naming the kind the figure
came from, and the summary totals them. With no config, no figure
appears anywhere.

## The bundled examples

Three deterministic rendered example calls live in `examples/autopsy/`
(seeded renderer, seed = `sha256(id)`, byte-identical on any machine) --
rendered demonstrations of the failure patterns, one each:

```bash
hotato autopsy examples/autopsy/audio/autopsy-01-barge-in-say-do.example.wav
hotato autopsy examples/autopsy/audio/autopsy-02-latency-dead-air.example.wav
hotato autopsy examples/autopsy/audio/autopsy-03-talk-over.example.wav
```

## From an incident to a regression gate

An incident worth keeping fixed graduates through the existing path:

```bash
hotato investigate call.wav # ranked candidates + the label command
hotato investigate label .hotato/investigate-state.json#1 --expect yield
hotato contract verify contracts/ --junit hotato.xml # the CI gate
```

## Scope and method

Deterministic energy measurement over time: per-frame RMS, a transparent
activity threshold, and the timing walk between the tracks. Timing and
floor-holding, not intent or transcription -- the scanner cannot know
whether a caller sound was "mhm" or "stop", and no accuracy percentage
appears anywhere. See [METHODOLOGY.md](../METHODOLOGY.md).
1 change: 1 addition & 0 deletions docs/LIFECYCLE.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ reproduces byte for byte.

| What | Command |
| :-- | :-- |
| One recording, zero config, the incident list | `hotato autopsy call.wav` |
| One recording, ranked candidate moments | `hotato investigate call.wav` |
| Every recent call on a stack, one sweep | `hotato sweep --stack vapi --since 7d` |
| A folder of recordings, ranked dashboard | `hotato analyze recordings/` |
Expand Down
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ New here? Start with **[GETTING-STARTED.md](GETTING-STARTED.md)**, then run

## Evaluate

- [AUTOPSY.md](AUTOPSY.md) - one recording, zero config: the incident list and report
- [INVESTIGATE.md](INVESTIGATE.md) - one recording in, ranked candidate moments out
- [ANALYZE.md](ANALYZE.md) - drop a folder, rank and hear the worst moments
- [ASSERTIONS.md](ASSERTIONS.md) - deterministic typed assertions over transcript, trace, and timing
Expand Down
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
19 changes: 19 additions & 0 deletions examples/autopsy/scenarios/autopsy-01-barge-in-say-do.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"id": "autopsy-01-barge-in-say-do",
"title": "Autopsy example: barge-in with a say-do gap (the agent keeps talking over the caller)",
"category": "autopsy-example",
"tags": ["autopsy", "barge-in", "talk-over", "example"],
"sample_rate": 16000,
"duration_sec": 12.0,
"note": "A deterministic rendered example call (seeded renderer, seed = sha256(id)) demonstrating the barge-in failure pattern for hotato autopsy: the caller takes the floor mid-utterance and the agent keeps talking over them past the search window, then stops on its own with no caller energy nearby.",
"transcript": {
"agent": "Our premium plan includes unlimited minutes, priority routing, and...",
"caller": "wait, I just need to cancel my appointment",
"agent_response": "I can cancel that appointment for you right now."
},
"reference_render": {
"continuous": true,
"agent_segments_sec": [[0.30, 6.50], [8.60, 12.0]],
"caller_segments_sec": [[3.00, 4.80], [7.40, 8.40]]
}
}
19 changes: 19 additions & 0 deletions examples/autopsy/scenarios/autopsy-02-latency-dead-air.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"id": "autopsy-02-latency-dead-air",
"title": "Autopsy example: dead air (the caller finishes and the agent answers seconds late)",
"category": "autopsy-example",
"tags": ["autopsy", "dead-air", "latency", "example"],
"sample_rate": 16000,
"duration_sec": 14.0,
"note": "A deterministic rendered example call (seeded renderer, seed = sha256(id)) demonstrating the dead-air failure pattern for hotato autopsy: the caller finishes a question and more than five seconds of silence pass before the agent's answer begins.",
"transcript": {
"agent": "Thanks for calling, how can I help?",
"caller": "what time does the pharmacy close on Sunday?",
"agent_response": "the pharmacy closes at six on Sunday"
},
"reference_render": {
"continuous": true,
"agent_segments_sec": [[0.30, 2.00], [9.50, 14.0]],
"caller_segments_sec": [[2.40, 4.20]]
}
}
18 changes: 18 additions & 0 deletions examples/autopsy/scenarios/autopsy-03-talk-over.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"id": "autopsy-03-talk-over",
"title": "Autopsy example: talk-over (the agent starts answering while the caller is mid-sentence)",
"category": "autopsy-example",
"tags": ["autopsy", "talk-over", "premature-start", "example"],
"sample_rate": 16000,
"duration_sec": 10.0,
"note": "A deterministic rendered example call (seeded renderer, seed = sha256(id)) demonstrating the talk-over failure pattern for hotato autopsy: the agent starts a fresh utterance while the caller still holds the floor and overlaps them well past a prompt-yield ceiling.",
"transcript": {
"caller": "I'd like to move my reservation to Friday at seven, and also...",
"agent": "Your reservation is confirmed for Thursday at six."
},
"reference_render": {
"continuous": true,
"agent_segments_sec": [[2.80, 10.0]],
"caller_segments_sec": [[1.00, 4.50]]
}
}
31 changes: 31 additions & 0 deletions examples/autopsy/scenarios/manifest.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
{
"scenarios": [
{
"id": "autopsy-01-barge-in-say-do",
"title": "Autopsy example: barge-in with a say-do gap (the agent keeps talking over the caller)",
"category": "autopsy-example",
"sample_rate": 16000,
"expected_yield": null,
"example_wav": "audio/autopsy-01-barge-in-say-do.example.wav",
"caller_wav": "audio/autopsy-01-barge-in-say-do.caller.wav"
},
{
"id": "autopsy-02-latency-dead-air",
"title": "Autopsy example: dead air (the caller finishes and the agent answers seconds late)",
"category": "autopsy-example",
"sample_rate": 16000,
"expected_yield": null,
"example_wav": "audio/autopsy-02-latency-dead-air.example.wav",
"caller_wav": "audio/autopsy-02-latency-dead-air.caller.wav"
},
{
"id": "autopsy-03-talk-over",
"title": "Autopsy example: talk-over (the agent starts answering while the caller is mid-sentence)",
"category": "autopsy-example",
"sample_rate": 16000,
"expected_yield": null,
"example_wav": "audio/autopsy-03-talk-over.example.wav",
"caller_wav": "audio/autopsy-03-talk-over.caller.wav"
}
]
}
1 change: 1 addition & 0 deletions examples/render_examples.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@
(os.path.join(HERE, "scenarios"), os.path.join(HERE, "audio")),
(os.path.join(HERE, "funnel-demo", "scenarios"), os.path.join(HERE, "funnel-demo", "audio")),
(os.path.join(HERE, "full-duplex", "scenarios"), os.path.join(HERE, "full-duplex", "audio")),
(os.path.join(HERE, "autopsy", "scenarios"), os.path.join(HERE, "autopsy", "audio")),
]


Expand Down
Loading
Loading