Skip to content

AGENTS.md

Latest commit

 

History

History
430 lines (314 loc) · 18.4 KB

AGENTS.md

File metadata and controls

430 lines (314 loc) · 18.4 KB
Raw

AGENTS.md

Unified guidance for AI coding assistants (Copilot, Claude, others) collaborating on the VideoAnnotator project.

1. Project Snapshot

  • Name: VideoAnnotator
  • Current Release: v1.5.0
  • In Progress: v1.6.0, the public release (docs/development/roadmap_v1.6.0.md). Sections below written for v1.2.x describe the registry's origins; CLAUDE.md and the roadmap are current.
  • Viewer: the Video Annotation Viewer lives in viewer/ (merged 2026-09-30); see §23.
  • Primary Users: Individual researchers (90%) needing simple local installs.
  • Secondary Users: Lab / enterprise research groups (10%).
  • Core Pillars: Modular pipelines, standardized outputs, reproducibility, minimal friction.

2. Primary Assistant Objectives

AI agents MUST optimize for:

  1. Low-friction local usability (no mandatory external services).
  2. Predictable, non-breaking incremental improvements (especially in v1.2.x series).
  3. Standard formats first (COCO, WebVTT, RTTM) – no bespoke schema proliferation.
  4. Future extensibility (registry metadata now → adaptive routing/plugins later).
  5. Clear, ASCII-safe console / log output (Windows encoding constraint: NO emoji, no fancy unicode blocks that may fail in PowerShell).

3. Output & Logging Rules

Concern Rule Rationale
Emoji / Unicode Prohibited in console/log prints Windows charmap crashes
Log Directory Use setup_videoannotator_logging(); do not reinvent handlers Central consistency
Structured Logs Prefer logger over print; CLI may still use typer.echo for UX Unified diagnostics
Error Prefix Use [ERROR], [WARNING], [OK], [START] Consistency + plain ASCII
JSON Output Provide optional --json for machine parsing (v1.2.1 introduction) Scripting support

4. Pipeline Registry (v1.2.1 Scope)

Minimal YAML-driven registry located at src/registry/metadata/ (to be added). Each file MUST define:

name: string (slug)
display_name: string
category: tracking|detection|analysis|segmentation|preprocessing
description: short sentence
outputs:
  - format: COCO|RTTM|WebVTT|JSON
    types: [list of semantic output types]
config_schema: { field_name: { type, default, description } }
examples: optional list (cli / api usage)
version: integer schema version

Assistant tasks involving pipelines should:

  • Load via registry once implemented rather than hard-coding.
  • Offer graceful degradation if metadata missing (warn, skip).
  • Avoid adding advanced fields early (resources, capabilities) unless explicitly approved (those are v1.3.0 targets).

5. API & CLI Conventions

Aspect Convention
CLI Base Command videoannotator (Typer app)
Job Submission videoannotator job submit
Pipelines Listing videoannotator pipelines --detailed (backed by registry)
Config Validation videoannotator config --validate path/to.yaml
Health Check /health (enriched incrementally)
Pipeline Endpoint /api/v1/pipelines dynamic (replace mock)

When adding new CLI commands:

  • Provide --json if the output lists entities.
  • Fail fast with exit codes; include a suggestion line for common errors.

6. Versioning & Consistency

  • Single version source in src/version.py (no duplicated literals).
  • Registry metadata includes its own version (schema evolution).
  • Generated docs (pipeline specs) MUST be reproducible – diffs trigger CI failure.
  • For v1.2.x the public API reports the package version; plan to introduce a dedicated API semantic version constant in v1.3.0.

7. Testing Expectations

Tier Path Goal Runtime
Unit tests/unit/ Fast logic validation < 1 min total
Integration tests/integration/ Cross-component behaviors ~5 min
Pipelines tests/pipelines/ Real or simulated model runs Longer

Assistant-created code MUST include at least minimal tests for:

  • Registry loader integrity.
  • API pipeline endpoint parity with registry.
  • Markdown generation (smoke: file contains each pipeline name).

7a. Development Environment & Terminal Commands

Workspace Configuration:

  • All terminals open in /workspaces/VideoAnnotator by default (dev container).
  • No need to prefix commands with cd /workspaces/VideoAnnotator &&.
  • Use uv for all Python operations (see section 21 for details).

Command Patterns:

# Correct (terminal is already in workspace)
uv run pytest tests/unit/

# Incorrect (unnecessary)
cd /workspaces/VideoAnnotator && uv run pytest tests/unit/

File Paths:

  • Use absolute paths when required by tools: /workspaces/VideoAnnotator/src/...
  • Relative paths work naturally since terminal CWD is the workspace root.

8. Deferred (Do NOT Implement in v1.2.1 Without Explicit Approval)

  • Plugin execution sandbox
  • Multi-modal correlation graph engine
  • Active learning feedback store
  • Real-time streaming/WebRTC stack
  • RBAC / SSO / multi-tenancy / audit log frameworks
  • Complex resource scheduling or autoscaling logic
  • Rich TUI or progress bar framework

9. Performance & Resource Principles

  • Favor lazy initialization; avoid loading heavy models outside initialize().
  • Avoid hidden network calls during import time.
  • Annotate any model download operations with explicit logging.
  • Provide timing logs (execution time context manager available in logging_config).

10. Security & Data Handling

  • Never log raw PII (e.g., full file system paths of user home beyond what’s already necessary).
  • Validate file existence before processing.
  • Reject unsupported video formats with clear error messages.
  • Avoid embedding secrets or tokens in code or docs.

11. Contribution Style for Agents

Action Required Practice
File Edits Small, atomic patches with rationale
New Features Add tests + docs stub in same PR scope
Refactors Preserve public CLI/API unless release notes updated
Docs Generation Provide script, never hand-edit generated file
Logging Additions Use existing logger categories (api, pipelines, requests, database)

12. Error & Validation Strategy

Standard error envelope (foundation for future):

{
  "error": {
    "code": "PIPELINE_NOT_FOUND",
    "message": "Pipeline 'xyz' not registered",
    "hint": "Run 'videoannotator pipelines --detailed' to list available pipelines"
  }
}

Assistant changes touching API endpoints should migrate ad hoc responses toward this structure (without breaking existing clients unless major release).

13. Windows Compatibility Checklist

Before approving output that touches user-visible text:

  • No emojis or fancy unicode
  • Paths use Path abstractions not hard-coded separators
  • Avoid color control chars (future optional feature)

14. Roadmap Alignment Matrix

Category v1.2.1 Expectation v1.3.0 Expansion
Registry Minimal YAML + loader Capability & resource aware, plugin aware
Config Validation Presence & primitive types Full schema + templates + precedence engine
Health Basic enrichment (registry, GPU flag) Deep performance & scaling metrics
Errors Simple envelope Taxonomy + GraphQL alignment
Docs Generated pipeline specs Marketplace + plugin publishing
CLI JSON option & consistent formatting Color/TUI, adaptive suggestions

15. Assistant Decision Heuristics

  1. Is the change needed for v1.2.1 scope? If not, defer.
  2. Can it be undone easily if incorrect? If not, request human confirmation.
  3. Does it introduce silent behavior change? If yes, add explicit note to changelog.
  4. Is there a testable seam? Add or extend a test BEFORE large refactor.

16. Typical Workflows for Agents

Add a New Pipeline (v1.2.1)

  1. Implement class in src/pipelines/<name>/
  2. Provide metadata YAML
  3. Update tests (registry presence)
  4. Regenerate docs (script) → commit

Extend Registry Field (v1.3.0-forward)

  1. Propose field (resources/capabilities) → doc it in metadata README
  2. Add parsing with backward-compatible default
  3. Update one metadata file as exemplar
  4. Add validation + tests

17. Common Anti-Patterns to Avoid

  • Adding model downloads in global import scope
  • Introducing new colorized/emoji output prematurely
  • Creating parallel config systems instead of extending registry
  • Hard-coding pipeline lists in multiple places
  • Adding advanced multi-modal logic in v1.2.1 timeframe

18. Quick Reference Checklist (Pre-Commit)

  • No emoji / unicode hazards
  • Registry unaffected OR metadata updated
  • Tests added/updated (if feature code)
  • Docs regenerated (if pipeline changes)
  • Logging consistent & non-duplicative
  • Version not duplicated outside version.py

19. Escalation & Deferral Tags (Suggest in PR Titles)

Use labels/phrases to keep scope controlled:

  • scope:v1.2.1-foundation
  • defer:v1.3.0
  • needs:registry-extension
  • type:docs-generation
  • type:cli-ux

20. Azure-Specific Guidance (If Applicable)

If generating Azure deployment or infra code:

21. Package & Environment Management (uv Mandatory)

The project STANDARD is to use uv (https://github.com/astral-sh/uv) for all Python dependency resolution, execution, and testing. Assistants MUST NOT rely on a pre-activated conda / venv environment when giving commands or running tests.

Required practices:

  1. Installation / Sync
  • To sync the environment exactly to pyproject.toml + uv.lock:
    • uv sync (never manually pip install inside the venv for project deps)
  1. Adding / Removing Dependencies
  • Add runtime dependency: uv add <package>
  • Add dev/test dependency: uv add --dev <package>
  • Remove: uv remove <package>
  • After changes: commit updated uv.lock.
  1. Running Code
  • Module / script: uv run python -m videoannotator ...
  • CLI entry (if defined): uv run videoannotator ...
  1. Tests & Quality
  • Full test suite: uv run pytest -q (or with markers/coverage flags)
  • Ruff lint (if configured): uv run ruff check .
  • Mypy (if configured): uv run mypy src/
  1. REPL / One-off
  • uv run python (never rely on external shell’s site-packages)
  1. Caching & Performance
  • Let uv manage wheels; do not manually clear caches unless diagnosing a build issue.
  1. Consistency Enforcement
  • If guidance or scripts previously referenced bare pytest / python, update them to uv run form.
  • Any test failure reproduction steps in issues/PR descriptions MUST use uv run.
  1. CI Alignment
  • CI should mirror: uv sync then uv run pytest .... Do not introduce alternative installers.

Common Anti-Patterns to Avoid (uv):

  • Using pip install -e . directly (use uv sync instead).
  • Activating a conda environment and running pytest without uv run.
  • Editing requirements.txt (the project is pyproject+lock driven; no requirements.txt unless generated artifact).

Assistant Heuristic Update:

  • Before running tests, always ensure commands are prefixed with uv run unless explicitly operating inside a scripted make target that already does so.
  • If a user asks for install instructions, default to uv commands.

Quick Reference:

uv sync                     # create/update environment
uv run pytest -q            # run tests
uv add requests             # add runtime dep
uv add --dev pytest-cov     # add dev dep
uv run videoannotator --help

If any discrepancy between an active shell environment and uv run behavior is suspected, treat the uv run result as canonical.

22. Pre-Commit & Type Checking Best Practices

The project uses pre-commit hooks (ruff, ruff-format, mypy, trailing-whitespace) that run automatically on git commit. Understanding their behavior reduces iteration cycles.

Type Checking Strictness Levels

The project has DIFFERENT mypy strictness for different modules (see pyproject.toml):

Strict modules (src/api/*, tests/api/*, tests/integration/test_api_integration):

  • check_untyped_defs = true
  • disallow_incomplete_defs = true
  • no_implicit_optional = true

Relaxed modules (pipelines, storage, utils, exporters, schemas):

  • Excluded from strict checking (reduces noise from ML/storage deps)

Common Type Issues & Solutions

  1. Pydantic Model Config Classes (RUF012)

    • Use json_schema_extra: ClassVar[dict[str, Any]] = {...}
    • Import: from typing import ClassVar, Any

  2. Exception Subclass Attributes

    • Use inline annotations in __init__: self.code: str = code
    • Avoid class-level annotations on Exception subclasses

  3. Mixed-Type Dictionaries

    • Declare explicit type upfront: detail: dict[str, str | int] = {}
    • Mypy infers dict value type from first assignment

  4. Mypy Cache Staleness

    • Clear cache when errors persist: rm -rf .mypy_cache ~/.cache/pre-commit/mypy*

Pre-Commit Workflow

Before committing:

uv run ruff format src/api/v1/    # Format first
uv run mypy src/api/              # Check types
uv run ruff check src/api/        # Lint

If pre-commit fails:

  • Auto-formatted files → re-stage: git add <files>
  • Mypy cache stale → clear: rm -rf .mypy_cache
  • After verifying fixes → last resort: git commit --no-verify

Hook order: trailing-whitespace → end-of-file-fixer → ruff → ruff-format → mypy

Files modified by first 4 hooks require re-staging.

API Module Requirements

For new src/api/ modules:

  • All parameters and returns typed
  • Pydantic Config: use ClassVar for class attributes
  • Exceptions: inline type annotations in __init__
  • Mixed dicts: explicit dict[str, str | int] hints
  • Import ClassVar, Any from typing

Quick Fixes

Error Fix
RUF012 mutable class attribute ClassVar[dict[str, Any]]
dict type mismatch Declare dict[str, str | int] upfront
Exception init type error Inline: self.attr: type = value
Stale mypy errors Clear .mypy_cache
Pre-commit formatting Re-stage files

23. Viewer (viewer/)

The Video Annotation Viewer: React 18 + TypeScript (strict), Vite, Tailwind, shadcn/ui, Zod. It renders VideoAnnotator's outputs (COCO person/face/OpenFace/VLM JSON, WebVTT, RTTM, scenes) over the video, and drives the API (job submission, SSE progress, results). It was a separate repository (InfantLab/video-annotation-viewer) until v1.5.0; it now shares VideoAnnotator's version number (viewer/package.json follows pyproject.toml; tests/unit/test_versions_match.py checks) and its changelog (the root CHANGELOG.md; viewer/CHANGELOG.md is frozen history). Standalone mode (drop in files, no server) is still supported.

Commands (Bun only; there is no npm lockfile)

bash scripts/dev.sh                      # API server + Vite dev server (port 19011, proxies to 18011)
cd viewer && bun run lint && bun run test:run
cd viewer && bun run typecheck           # zero errors, checked in CI (plain `tsc` checks nothing)
bash scripts/gen_viewer_api_types.sh    # after an API change: regenerate src/api/schema.d.ts
cd viewer && bun run e2e                 # Playwright smoke (bun run e2e:install first)
bash scripts/build_viewer.sh             # rebuild into src/videoannotator/viewer_static/

Any change under viewer/src/ needs scripts/build_viewer.sh and the rebuilt viewer_static/ committed in the same commit; CI's viewer job runs --check and fails on a stale bundle.

Where things go

  • API client: viewer/src/api/client.ts; hooks (SSE, health, job actions): viewer/src/hooks/; pages: viewer/src/pages/; shared types: viewer/src/types/annotations.ts.
  • A new output format: type in types/annotations.ts, parser in lib/parsers/<format>.ts, detection and merge in lib/parsers/merger.ts, drawing in components/VideoPlayer.tsx (and the timeline), validation in lib/validation.ts, tests in viewer/src/test/.
  • Changing what a pipeline writes, or how the viewer reads it: refresh the real-output fixtures in tests/fixtures/viewer_contract/ (its README says how) so viewer/src/test/contract/videoannotator-outputs.test.ts tests the new shape.
  • Job results load from the artifacts zip (hooks/useZipDownloader.ts), through the same detection and merge as dropped-in files.

Conventions and pitfalls

  • Reuse shadcn/ui components (viewer/src/components/ui/); validate external data with Zod.
  • ESLint's react-refresh/only-export-components: a module that exports a component exports only components; move constants, helper hooks and variants to their own modules.
  • Keep react-hooks/exhaustive-deps clean (don't disable it); stabilise derived deps with useMemo.
  • Use 127.0.0.1, not localhost, for the API: the browser keeps separate storage for the two, so a token saved under one is missing under the other.
  • Tests mock the network; no real API calls in CI. jsdom's File drops Node Buffers (size 0): build test files from strings.
  • Large files: load progressively; don't block the UI.
  • Viewer specs before the merge are in viewer/specs/ (history); new viewer work gets a spec in specs/ like any other feature, under the one constitution (.specify/memory/constitution.md), which covers the viewer since v1.1.0: no telemetry (I), the formats it reads (II), layers (IV), UI stability (V), and faithful display (VI): never create or hide values; holding the latest sample until the next is allowed; overlays attributable to their pipeline.