Skip to content

feat(runtime): add customer-apps consumption command#236

Merged
cdot65 merged 4 commits into
cdot65:mainfrom
scthornton:feat/runtime-customer-apps-consumption
May 28, 2026
Merged

feat(runtime): add customer-apps consumption command#236
cdot65 merged 4 commits into
cdot65:mainfrom
scthornton:feat/runtime-customer-apps-consumption

Conversation

@scthornton

Copy link
Copy Markdown
Contributor

Surfaces the SCM "AI Security > Runtime > API Applications" detail panel through the CLI, sourced from the new mgmt.dashboard SDK namespace.

Closes #222.

Status: DRAFT — pending SDK release

This PR consumes a new SDK API (mgmt.dashboard.application + mgmt.dashboard.applicationViolationBreakdown) introduced in cdot65/prisma-airs-sdk#175. CI on this branch will not pass until that PR merges, the SDK ships (proposed 0.11.0), and the @cdot65/prisma-airs-sdk dep here is bumped to the new minor. The implementation has been fully exercised end-to-end via pnpm link against the local SDK source and a live tenant — see the test plan below.

Command

airs runtime customer-apps consumption [appName]
  [--time-interval 7|30|60]      (default 30)
  [--output pretty|table|csv|json|yaml]  (default pretty)

appName is optional — omit it to loop every customer app in the tenant. The appId is resolved internally from customer-apps list.

Live evidence — single app (pretty)

$ airs runtime customer-apps consumption chatbot

  chatbot  (5e16929a-4186-4937-875b-9e84473cf3df)
    Monitoring since: 2026-04-29T17:04:52Z
    Source:           api
    Cloud:            other
    Profiles:         ms-tuned, golden-v2, ...

    Token consumption:
      Daily avg:     744.233K
      Monthly total: 17.713755M

    Sessions:
      Total:     56936
      Violating: 31137

    Detectors (59959 violating, 8/10 firing):
      agent_security       11174  c=0 h=0 m=11174 l=0
      dlp                    219  c=0 h=0 m=53 l=166
      malicious_code           4  c=0 h=4 m=0 l=0
      pi                   11230  c=0 h=0 m=11230 l=0
      source_code              7  c=0 h=7 m=0 l=0
      tc                    8414  c=0 h=0 m=4326 l=4088
      topic_guardrails     28501  c=0 h=0 m=28501 l=0
      uf                     410  c=0 h=0 m=0 l=410

Live evidence — CSV (one row per detector, app-level context repeated)

$ airs runtime customer-apps consumption chatbot --output csv --time-interval 60
App,AppId,MonitoringSince,DailyAvg,MonthlyTotal,Sessions,Violating,Detector,C,H,M,L,Total
chatbot,5e16929a-...,2026-04-17T16:11:47Z,526.493K,17.713755M,57157,31300,agent_security,0,0,11176,0,11176
chatbot,5e16929a-...,2026-04-17T16:11:47Z,526.493K,17.713755M,57157,31300,contextual_grounding,0,0,0,0,0
...

Self-contained rows so the output can pipe directly into spreadsheets / BigQuery / etc. without join logic.

Error paths (also exercised live)

$ airs runtime customer-apps consumption no-such-app
  Error: Customer app not found: "no-such-app". Run \`airs runtime customer-apps list\` to see available apps.

$ airs runtime customer-apps consumption chatbot --time-interval 14
  Error: --time-interval must be 7, 30, or 60 (the API rejects other values)

The API enforces time_interval as an enum (verified live: only 7, 30, 60 return 200; everything else returns 400). Client-side validation rejects bad values before incurring a request.

Design notes

  • Two endpoints, one command: mgmt.dashboard.application (overview + tokens) and mgmt.dashboard.applicationViolationBreakdown (per-detector counts) are called in parallel and merged into a single normalized CustomerAppConsumption shape.
  • AppId resolution: The dashboard endpoints require both appId AND appName. The CLI only takes a name from users, then looks up the UUID via customer-apps list. This costs one extra API call per command invocation but keeps the UX simple.
  • Renderer: pretty mode is for humans (sections, colors, only firing detectors). Structured modes (table/csv/json/yaml) emit one row per detector per app with full app-level context on every row — optimized for piping into reporting tools.
  • All-apps mode: errors on individual apps are caught and reported per-app so one bad app doesn't kill the whole loop.

Test plan

  • pnpm build (tsc)
  • pnpm lint (biome — only pre-existing warnings in test/walk.spec.ts unrelated to this PR)
  • pnpm test (712/712 — adds 4 new tests for getCustomerAppConsumption)
  • Live single-app: consumption chatbot
  • Live all-apps loop (5 apps in tenant, all round-trip) ✓
  • Live CSV/JSON output ✓
  • Live error: bad app name ✓
  • Live error: invalid --time-interval (14) ✓
  • Live alternate window: --time-interval 60 ✓
  • CI on this branch — pending SDK 0.11.0 release + dep bump

Out of scope (separate follow-ups)

scthornton and others added 3 commits May 28, 2026 09:26
…olations

Surfaces the SCM "AI Security > Runtime > API Applications" detail panel
through the CLI, sourced from the SDK's new mgmt.dashboard namespace:

  airs runtime customer-apps consumption [appName]
    [--time-interval 7|30|60]
    [--output pretty|table|csv|json|yaml]

Single-app mode renders a pretty section with token consumption (with K/M
scale qualifiers), session counts, and firing detectors. Omitting appName
loops every customer app in the tenant. Structured outputs emit one row
per detector per app so downstream tools can pivot without join logic.

Notes
- Resolves appId from `customer-apps list` internally — users only pass
  the human-readable name. Throws with guidance if the name isn't found.
- Validates --time-interval client-side against the API's enum
  (7, 30, 60); other values are rejected before incurring a request.
- Per-detector rows include monitoring_since so the data is self-contained
  for chargeback reporting.

Closes cdot65#222. Requires the new mgmt.dashboard SDK API
(cdot65/prisma-airs-sdk PR pending) — bump @cdot65/prisma-airs-sdk after
that lands and ships.

@cdot65 cdot65 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validation 2026-05-28 (post-SDK 0.11.0)

SDK PR cdot65/prisma-airs-sdk#175 merged 17:05Z; @cdot65/prisma-airs-sdk@0.11.0 published. Bumped dep, regenerated customer-apps.md reference, allowlisted runtime customer-apps consumption pending curated sidecars. Exercised end-to-end against a 6-app tenant (substituted Insomnia/CLI for chatbot — no chatbot in this tenant).

Results:

  • Single app pretty output renders as described (header, monitoring window, token block, sessions, detector breakdown with severity counts)
  • --time-interval accepts 7/30/60, rejects 14 with client-side message --time-interval must be 7, 30, or 60 (the API rejects other values)
  • CSV one-row-per-detector with app context repeated — 7 rows for the 7-detector profile on Insomnia
  • JSON / YAML structured output — same per-detector shape, app metadata repeated per record
  • Table output renders cleanly
  • All-apps loop completes (6 apps in tenant, all 6 processed, zero-traffic apps continue past silently — no early kill)
  • Not-found error renders Customer app not found: "no-such-app". Run airs runtime customer-apps list to see available apps.
  • All CI gates green post-bump (lint, typecheck, test, docs-check, docs-build)

Observations / asks

  1. Sidecars still owed. I allowlisted runtime customer-apps consumption in docs/cli/examples/.missing-allowlist to unblock CI; please replace that allowlist entry with curated sidecars in docs/cli/examples/runtime.yaml before merge. Phase 3 output above is reusable as fixture material.
  2. Zero-traffic apps emit no rows in structured formats. When an app has 0 sessions, --output csv|json|yaml returns an empty body (no header row in CSV, empty array in JSON). Pretty renders no detector violations in window for the same case. Consider emitting a single empty-state row in structured formats so downstream consumers don't have to special-case the no-data path. Not a blocker.
  3. The mechanical docs:gen regen of docs/cli/runtime/customer-apps.md was missing from the PR (likely a missed pnpm docs:gen before commit).

LGTM on the feature — ready for @scthornton's merge once sidecars are added.

Captures 8 examples (pretty/table/csv/json/yaml + 60-day window +
all-apps loop + both error paths) from live tenant; removes the
allowlist entry; regenerates customer-apps.md so the reference page
now renders real input/output instead of "Example needed".

Addresses follow-up cdot65#1 from the PR cdot65#236 validation comment.
@cdot65 cdot65 marked this pull request as ready for review May 28, 2026 17:58
@cdot65 cdot65 merged commit 49bbc33 into cdot65:main May 28, 2026
5 checks passed
@cdot65 cdot65 mentioned this pull request May 28, 2026
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: surface per-app token consumption + violation breakdown (SCM dashboard mgmt API)

2 participants