feat(telemetry): restore delta metrics, add delta_function_count & active-developer signal

[anthonyiscoding]

Summary

Fixes items 4 & 5 from III-3. Re-enables the delta metrics that were commented out in #1390 and extends them so Amplitude can cleanly separate active development from passive runtime (the Rohit-agentmemory problem).

Tracking-plan diff

Event type: heartbeat and engine_stopped (anonymous Amplitude stream).

Added

Property	Type	Description
delta_function_count	int (signed)	Change in registered user-function count since last heartbeat. Negative if functions were removed.
delta_trigger_count	int (signed)	Change in registered trigger count.
delta_worker_count	int (signed)	Change in connected worker count.
delta_function_registrations	uint	Cumulative-counter delta — catches hot-reload / edit cycles where the function id stays the same.
delta_trigger_registrations	uint	Same, for trigger registrations.
functions_added	string[]	Function ids that appeared since last heartbeat.
functions_removed	string[]	Function ids that disappeared.
workers_added	string[]	Worker descriptors (runtime:framework) that appeared.
workers_removed	string[]	Worker descriptors that disappeared.
is_active	bool	delta_invocations_total > 0 — runtime activity this period.
is_active_developer	bool	Any of: functions added/removed, re-registration, or signed-count change ≠ 0.

Restored (were commented out in #1390)

delta_invocations_total, delta_invocations_success, delta_invocations_error, delta_api_requests, delta_queue_emits, delta_queue_consumes, delta_pubsub_publishes, delta_pubsub_subscribes, delta_cron_executions.

Deprecated / Changed

None. All additions.

Cohort definition (Amplitude)

Active Developer cohort: users who sent a heartbeat where is_active_developer = true AND session_start = false in the last 28 days.

• is_active_developer is always false on the session-start heartbeat by design (first snapshot has no prior state to diff against), so this filter keeps the cohort clean and excludes the inevitable boot-time registration noise.
• Active Runner cohort (complement): is_active = true AND NOT in the Active Developer cohort. Captures the agentmemory-style population — people running iii as an embedded runtime without touching the code.

Design notes

• DeltaAccumulator is now shared between the session-start spawn and the interval loop spawn via Arc<Mutex<_>>, so the loop's first delta is relative to the session-start snapshot rather than re-reading zero. The original commented-out code created two separate accumulators, which would have triple-counted activity across the session-start → first-loop-tick window.
• State-change deltas (delta_function_count, functions_added, etc.) are deliberately zeroed on the first snapshot so session start is not misclassified as active development.
• is_active_developer intentionally excludes pure worker churn (SDK reconnects with the same function set) — those don't indicate code edits.

Test plan

• cargo test -p iii --lib workers::telemetry:: — 151 tests pass (7 new tests cover add/remove/stable/reregistration/worker-churn/first-snapshot/schema).
• cargo fmt -p iii --check clean.
• cargo clippy -p iii --lib — no new warnings in workers/telemetry.
• Verify in Amplitude after rollout: heartbeat events include new properties; Active Developer cohort populates with expected internal users; Active Runner cohort stays stable across idle periods.

Related

Findings for items 1–3 (project_name reporting, container/user conflation, per-user worker listing) are in the comment on III-3. They surface follow-up fixes (container device-id instability, hardcoded Amplitude key, absence of a project_name directory fallback) that warrant separate PRs so reviewers can consider each independently.

Summary by CodeRabbit

• Tests

  • Added comprehensive test coverage validating metrics accuracy, state change detection, activity pattern recognition, and system behavior verification.

• Refactor

  • Enhanced telemetry infrastructure with improved delta metrics computation, refined state tracking capabilities, and more accurate heartbeat cycle data collection.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
iii-website	Ready	Preview, Comment	Apr 23, 2026 8:55pm

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 95734380-1672-4fed-8b84-25478e251abe

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The file replaces commented-out delta metrics code with a working DeltaAccumulator/DeltaSnapshot implementation that computes deltas per heartbeat by snapshotting cumulative metric counters. New derived boolean fields is_active and is_active_developer are added. The TelemetryWorker owns a shared accumulator and injects delta fields into heartbeat event payloads. Extensive tests validate the implementation.

Changes

Cohort / File(s)	Summary
Delta Metrics Implementation engine/src/workers/telemetry/mod.rs	Replaces commented-out code with functional DeltaAccumulator and DeltaSnapshot types. Implements delta computation per heartbeat using saturating subtraction, state diffing for function/worker ID sets, and explicit handling of first snapshot behavior to emit cumulative deltas while zeroing state changes.
Active Status Derivation engine/src/workers/telemetry/mod.rs	Adds derived boolean fields is_active (based on invocations count) and is_active_developer (computed from function set changes, registration deltas, excluding worker churn) to track engagement metrics.
Telemetry Worker Integration engine/src/workers/telemetry/mod.rs	Updates TelemetryWorker to own a shared, mutex-protected accumulator and injects computed delta fields into both boot and periodic heartbeat event property payloads.
Test Coverage engine/src/workers/telemetry/mod.rs	Adds extensive test suite validating first-snapshot behavior, added/removed function detection, reregistration detection, invocation gating, worker churn behavior, and JSON key emission.

Sequence Diagram

sequenceDiagram
    participant HB as Heartbeat Event
    participant TW as TelemetryWorker
    participant DA as DeltaAccumulator
    participant EP as Event Payload
    
    HB->>TW: Trigger periodic heartbeat
    TW->>DA: Request snapshot of current state
    DA->>DA: Snapshot metric counters & engine state
    DA->>DA: Compute deltas using saturating_sub
    DA->>DA: Compute is_active & is_active_developer
    DA-->>TW: Return DeltaSnapshot
    TW->>EP: Inject delta fields into payload
    TW->>HB: Emit enriched heartbeat event

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Suggested reviewers

• ytallo

Poem

🐰 Deltas dance through every beat,
Snapshots capture metrics sweet,
Accumulators hoard the state,
Active flags we calculate,
Telemetry logs all complete! ✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main changes: restoring delta metrics and adding the delta_function_count and is_active_developer signal.
Description check	✅ Passed	The description comprehensively covers the What, Why, and implementation details with extensive tracking-plan diffs, design notes, and test plan, though the required template sections are not explicitly labeled.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent/telemetry-engineer/2f2d1c02

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

engine/src/workers/telemetry/mod.rs (1)

891-938: ⚠️ Potential issue | 🟡 Minor

Startup ordering: periodic task can beat the boot task to the first snapshot when heartbeat_interval_secs is small.

The shared Arc<Mutex<DeltaAccumulator>> means whichever task calls snapshot() first gets the "first snapshot" zeroing behavior. The boot task waits up to 120 s (or until first_user_invocation); the periodic task consumes an immediate first interval.tick().await at line 944 and then waits interval_secs before the next tick.

If interval_secs < ~120s and no user invocation arrives promptly, the periodic heartbeat (session_start=false) will take snapshot #1, and the boot heartbeat (session_start=true) will get snapshot #2 with non-zero state deltas relative to whatever the periodic task captured. In production this is unlikely (default heartbeat_interval_secs = 6 * 60 * 60), but it's reachable via TelemetryConfig.heartbeat_interval_secs and is exactly the configuration exercised by test_telemetry_module_background_tasks_and_destroy_run_without_network (interval = 1 s).

Combined with the function_registrations > 0 issue in the comment above, this means a short-interval configuration could emit a periodic heartbeat with session_start=false and is_active_developer=true purely from session start.

Options:

• Prime the accumulator synchronously in TelemetryWorker::create (before either task spawns) so both tasks always see initialized=true.
• Or have the periodic task await a signal from the boot task before taking its first snapshot.
• Or, as suggested above, decouple is_active_developer from first-snapshot cumulative counters so ordering doesn't matter.

Also applies to: 940-1002

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@engine/src/workers/telemetry/mod.rs` around lines 891 - 938, The race happens
because both the periodic heartbeat task and the boot task call
DeltaAccumulator::snapshot (via delta_for_started.lock().snapshot(&snap)), so
whichever runs first gets the “first snapshot” zeroing; fix this by priming the
accumulator synchronously in TelemetryWorker::create before spawning either
background task: call collect_engine_snapshot once and invoke
delta_accumulator.lock().expect(...).snapshot(&initial_snap) (DeltaAccumulator)
so initialized=true holds for both the periodic (interval.tick) task and the
boot task; update references to collect_engine_snapshot,
TelemetryWorker::create, and DeltaAccumulator (delta_for_...) to reflect the new
pre-spawn initialization.

🧹 Nitpick comments (1)

engine/src/workers/telemetry/mod.rs (1)

2598-2806: Test coverage is thorough — consider one more case for startup ordering.

The seven new tests cover first-snapshot zeroing, added/removed diffing, stable sets, reregistration via track_function_registered(), is_active invocation gating, worker churn isolation from is_active_developer, and key emission. Nice.

One gap worth closing: none of the tests actually exercise the production path where function_registrations is non-zero before the first snapshot. Adding a test that pre-increments collector().function_registrations (or calls track_function_registered()) before the first acc.snapshot(&snap) would lock down the intended semantics for is_active_developer on the first snapshot (see the comment on lines 470–480 above). It would also make the test reflect real engine boot behavior rather than the artificial globals-at-zero scenario set up by reset_telemetry_globals().

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@engine/src/workers/telemetry/mod.rs` around lines 2598 - 2806, Add a new
serial test that verifies function_registrations set before the first snapshot
still prevents the first-snapshot-from-looking-like-active-dev: call
reset_telemetry_globals(), then call collector::track_function_registered() (or
increment the global function_registrations) before creating
DeltaAccumulator::new() and invoking acc.snapshot(&snap) (use
make_engine_snapshot like other tests); then assert that the first returned
delta has function_registrations == 1 and that is_active_developer() is true.
Ensure the test name reflects startup_registration_before_first_snapshot (or
similar) and follows the same setup/teardown pattern as the other tests.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@engine/src/workers/telemetry/mod.rs`:
- Around line 470-480: is_active_developer() currently returns true if
function_registrations>0 which can make the very first snapshot report developer
activity; fix by making first snapshot ignore cumulative registration
counters—add a first_snapshot boolean (set from !self.initialized in the
DeltaAccumulator snapshot creation) to DeltaSnapshot and have
is_active_developer() return false when first_snapshot is true (or alternatively
zero function_registrations/trigger_registrations for that first snapshot);
update references to function_registrations and trigger_registrations in
is_active_developer and ensure DeltaAccumulator sets first_snapshot before
mutating initialized.
- Around line 482-560: The new telemetry properties added in insert_into
(delta_invocations_total, delta_invocations_success, delta_invocations_error,
delta_api_requests, delta_queue_emits, delta_queue_consumes,
delta_pubsub_publishes, delta_pubsub_subscribes, delta_cron_executions,
delta_function_registrations, delta_trigger_registrations, delta_function_count,
delta_trigger_count, delta_worker_count, functions_added, functions_removed,
workers_added, workers_removed, is_active, is_active_developer) must be
documented and consumers updated: add a telemetry event schema entry describing
each property's type, meaning and example usage in the project telemetry docs
(include the Active Developer cohort recipe: filter is_active_developer == true
AND session_start == false), update any sdk consumer code to accept the new
properties and confirm they expect workers_added/workers_removed format
"{runtime}:{framework}" or "{runtime}", and validate Amplitude ingestion rules
accept the specified integer types (i64 for deltas vs u64 for cumulative) before
rolling out.

---

Outside diff comments:
In `@engine/src/workers/telemetry/mod.rs`:
- Around line 891-938: The race happens because both the periodic heartbeat task
and the boot task call DeltaAccumulator::snapshot (via
delta_for_started.lock().snapshot(&snap)), so whichever runs first gets the
“first snapshot” zeroing; fix this by priming the accumulator synchronously in
TelemetryWorker::create before spawning either background task: call
collect_engine_snapshot once and invoke
delta_accumulator.lock().expect(...).snapshot(&initial_snap) (DeltaAccumulator)
so initialized=true holds for both the periodic (interval.tick) task and the
boot task; update references to collect_engine_snapshot,
TelemetryWorker::create, and DeltaAccumulator (delta_for_...) to reflect the new
pre-spawn initialization.

---

Nitpick comments:
In `@engine/src/workers/telemetry/mod.rs`:
- Around line 2598-2806: Add a new serial test that verifies
function_registrations set before the first snapshot still prevents the
first-snapshot-from-looking-like-active-dev: call reset_telemetry_globals(),
then call collector::track_function_registered() (or increment the global
function_registrations) before creating DeltaAccumulator::new() and invoking
acc.snapshot(&snap) (use make_engine_snapshot like other tests); then assert
that the first returned delta has function_registrations == 1 and that
is_active_developer() is true. Ensure the test name reflects
startup_registration_before_first_snapshot (or similar) and follows the same
setup/teardown pattern as the other tests.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 5713e935-d33d-473c-8b82-ccc3342a5b91

📥 Commits

Reviewing files that changed from the base of the PR and between cd7eb05 and 1c092d4.

📒 Files selected for processing (1)

• engine/src/workers/telemetry/mod.rs