Local video scene intelligence for NVIDIA DGX Spark and Apple Silicon. ScreenLens extracts meaningful frames from screen recordings, captions or transcribes them with a local multimodal model, embeds them with OpenCLIP, indexes them in ChromaDB, and reconstructs visible code, documents, PDFs, and GUI walkthroughs. Linux/ARM64 uses vLLM + CUDA by default; Apple Silicon uses oMLX + MPS. Ollama remains an optional fallback.
graph TD
A[Video Input<br/>.mov / .mp4] --> B[Hybrid Keyframe Detection<br/>SSIM + pHash + HSV]
B --> C{Local vision backend}
C -->|DGX Spark| D[vLLM<br/>NVIDIA Qwen3.6 NVFP4]
C -->|Apple Silicon| E[oMLX<br/>configured multimodal model]
C -->|optional| F[Ollama vision]
D --> G[Captions / Verbatim OCR]
E --> G
F --> G
G --> H[OpenCLIP Embeddings<br/>CUDA / MPS / CPU]
H --> I[ChromaDB Search]
G --> J[LangGraph Reconstruction<br/>plan → workers → QA]
J --> K[Reconstructed Artifacts<br/>data/*/output]
The vLLM and oMLX paths share the same small OpenAI-compatible client, including image data URLs and chat_template_kwargs.enable_thinking=false for Qwen-style models. Platform detection changes defaults, not the pipeline itself.
| Component | Technology | Purpose |
|---|---|---|
| Frame extraction | Hybrid SSIM + pHash + HSV detection | Capture distinct screens while bounding long static intervals |
| DGX vision/text | nvidia/Qwen3.6-27B-NVFP4 via vLLM |
Caption frames, perform OCR, summarize, and reconstruct through one dense local model |
| Apple vision/text | Configured multimodal model via oMLX | Native Apple Silicon OpenAI-compatible inference |
| Optional fallback | Ollama | Alternative captioning and summarization when explicitly selected |
| Visual embeddings | OpenCLIP ViT-B-32 | Semantic image/text vectors on CUDA, MPS, or CPU |
| Vector storage | ChromaDB | Persistent per-video semantic search |
| Reconstruction | LangGraph | Classify recordings and rebuild code/docs/demo artifacts with QA retries |
| Interfaces | Typer + Rich, web command deck | CLI and browser GUI |
| Host | Inference | Embeddings | Concurrent image requests |
|---|---|---|---|
| Linux ARM64 / DGX Spark | vLLM | CUDA | 2 |
| Darwin ARM64 / Apple Silicon | oMLX | MPS | 4 |
| Other hosts | oMLX unless overridden | CPU | 4 |
Caption output is capped at 32,768 tokens by default. The bundled dense Spark service exposes a 262,144-token total context, leaving ample prompt and image-token headroom while reconstruction can use the full served window.
The checked helper creates an isolated Python 3.12 CUDA environment, validates the host and OpenCLIP, starts or reuses the exact vLLM model, and performs a real vision smoke test:
(umask 077; touch .env)
chmod 600 .env
${EDITOR:-nano} .env # add HF_TOKEN=hf_... if ScreenLens must start vLLM
./setup_and_run_dgx.sh doctor
./setup_and_run_dgx.sh setup
./setup_and_run_dgx.sh llm-up
./setup_and_run_dgx.sh llm-wait
./setup_and_run_dgx.sh smoke
./setup_and_run_dgx.sh runrun launches the web command deck (serve --no-browser) with no arguments or passes a CLI command through:
./setup_and_run_dgx.sh run ingest input-videos/demo.mov
./setup_and_run_dgx.sh run transcribe input-videos/demo.movDigitalTwin uses the same model and loopback port. The helper reuses an already-ready exact-model service instead of starting a conflicting container. See the complete DGX Spark guide for CUDA wheel pins, cache sharing, memory sizing, service ownership, and troubleshooting.
Start oMLX on http://127.0.0.1:8000/v1, configure MLX_* or OMLX_* values in .env, then use the existing Conda launcher:
./setup_and_run_macos.sh
./setup_and_run_macos.sh ingest "video.mov"The script creates a screenlens Conda environment with Python 3.11 and ffmpeg, installs the package in editable mode, preserves an existing .env, and defaults to the web command deck (serve). Set SCREENLENS_CONDA_ENV to choose another environment name. On Linux/ARM64 it exits with directions to the checked Spark helper so a generic pip install cannot replace CUDA 13 wheels with CPU wheels.
pip install -e ".[dev]"
pytest tests/ -vInstall ffmpeg separately and ensure the selected inference server is running. Ollama is not required for the normal vLLM or oMLX paths.
Commands that contact the direct inference service accept provider-neutral flags:
python -m src.cli ingest video.mov \
--backend vllm \
--inference-url http://127.0.0.1:8000/v1 \
--inference-model nvidia/Qwen3.6-27B-NVFP4 \
--inference-api-key local \
--device cuda \
--batch-size 2--vllm-url, --vllm-model, and --vllm-api-key are provider-specific aliases; the existing --omlx-* spellings remain compatible. SCREENLENS_BACKEND, SCREENLENS_DEVICE, and SCREENLENS_BATCH_SIZE override platform defaults. Provider credentials and models can also come from VLLM_*, MLX_*, or OMLX_* variables as appropriate.
ScreenLens drives two roles against one OpenAI-compatible endpoint:
| Role | Used by | Requirement | oMLX default |
|---|---|---|---|
| vision | captioning, verbatim OCR | must be vision-capable — a text-only model answers "no image provided" for every frame | Qwen3.6-27B-bf16 (OCR_MODEL / MLX_MODEL) |
| text | summaries, reconstruction plan/QA, transcript cleanup | never sees an image; must stay coherent on markdown with indented lists, which is what captions are | DeepSeek-V4-Flash-0731-MXFP4-MLX (LLM_MODEL) |
On DGX Spark both resolve to the single served vLLM checkpoint. On Apple
Silicon the defaults split the roles — Qwen3.6 sees, DeepSeek-V4-Flash reasons
— and they resolve independently, so pointing LLM_MODEL at
Qwen3.6-27B-bf16 to keep one resident checkpoint is fully supported.
The DeepSeek text default is guarded against the family's known failure mode
on ScreenLens inputs. DeepSeek-V4-Flash-0731-MLX (the older 8-bit build)
under oMLX emits its BOS token until the token budget runs out when a
multi-sentence system prompt accompanies a markdown list item with a leading
indent (\n - ), which every caption has. ScreenLens folds the system
prompt into the user turn for deepseek-v4 ids — the verified fix — and
detects any output that still degenerates, raising InferenceDegenerateError
rather than saving it as a summary, so a failure is loud rather than silent.
If you see that error, fall back to Qwen3.6-27B-bf16 for the text role.
Check what your endpoint serves and which side of the vision line each model falls on:
python -m src.cli modelsThe capability guard aborts a transcribe run whose OCR model is text-only,
and the command deck's vision selector only offers models that can see.
# Uses vLLM/CUDA on DGX Spark or oMLX/MPS on Apple Silicon.
python -m src.cli ingest "video.mov"
# Optional Ollama captioning fallback.
python -m src.cli ingest "video.mov" --backend ollama --strategy fixed_fps --fps 1
# Search one or all timestamped collections and synthesize an answer locally.
python -m src.cli search "What application is shown?" --data-dir ./data --top-k 5
# Ingest and search in one command.
python -m src.cli run "video.mov" "Summarize the workflow"Search summarization follows the configured platform backend; it does not require Ollama unless Ollama was explicitly selected. Each ingestion creates data/<stem>_<YYYYMMDD_HHMMSS>/ with independent frames, captions, and ChromaDB data.
Re-running a pipeline on the same video resumes the newest prior run instead of paying for the same work twice: ingestion reuses the extracted frames, any per-frame captions already written, and a populated vector store; transcription reuses cached OCR. Resume only engages when the run's stored video path/size and extraction config still match — pass --fresh to force a new run folder (supported by ingest, run, batch, and transcribe; the web deck resumes automatically).
Repeated work is avoided at other layers too: the CLIP embedder is a process-wide shared instance, so the web deck and multi-collection searches load OpenCLIP once instead of per query; search and full-video summaries are cached per run folder in summary_cache.json (keyed by model + prompt content); captioning keeps all pending frames in flight in a single pool with batch_size workers rather than stalling at each chunk boundary; and verbatim OCR sends frames to the server as in-memory JPEG q92 re-encodes — on-disk frames stay lossless PNG — shrinking multi-MB payloads roughly 5–10×.
python -m src.cli batch input-videos/
python -m src.cli summarizeBatch ingestion gives every video its own timestamped directory. Full-video summary reads all stored captions and chooses a single-pass or hierarchical strategy based on the selected backend's configured context window.
python -m src.cli reconstruct
python -m src.cli assembleReconstruction classifies each recording, plans the artifact work, processes it deterministically, and runs up to three QA iterations before saving under data/<slug>/output/. Completed folders are skipped, Pass-1 segment notes are persisted to output/segment_notes.json so an interrupted run resumes cheaply, and the planning generation is cached across QA retries. Assembly can combine per-recording outputs into one project tree under OUTPUT/.
# Discover served models and their likely vision capability.
python -m src.cli models
# OCR frames, stitch scrolling overlap, and preserve raw text.
python -m src.cli transcribe input-videos/policies.mov
# Optional seam/indent cleanup; guarded against dropped content.
python -m src.cli transcribe input-videos/document.mov --cleanup
# Apple-only deterministic Vision cross-check for code.
python -m src.cli transcribe input-videos/code.mov --deterministicThe transcribe path copies visible text rather than describing it. A live image probe rejects blind/text-only deployments before a full run. Cleanup is off by default; when enabled, a per-chunk coverage guard keeps the raw stitched chunk whenever the model drops content. Outputs are transcript.raw.md, transcript.md, ocr/all_ocr.json, and metadata under the timestamped data directory.
python -m src.cli serve # http://127.0.0.1:8760, opens a browser
python -m src.cli serve --port 9000 --no-browser
python -m src.web # same thing without TyperA local dashboard over every pipeline: run register, frame gallery served
straight off disk, semantic search, pipeline control, and an artifact reader.
It is stdlib http.server plus one static page — no framework, no build step,
and no dependency beyond what ScreenLens already installs.
It binds loopback-only and refuses non-loopback clients on every /api/ route,
because it starts jobs and reads frames off disk. One job runs at a time: the
pipelines share a single model endpoint and one CLIP device.
There is no stop button. A running job cannot be cancelled from the page:
interrupting it would mean aborting the in-flight HTTP request to the model
server, which the stdlib client cannot do mid-read. Jobs run as daemon threads,
so stopping the deck (Ctrl-C) ends them — the model server may keep generating
the current request for a while afterwards. This matters more than it sounds:
reconstruct over a large caption set can run for hours (see Performance
Notes), so check the run size before starting one.
The header shows both model roles at once. If the configured vision model is not vision-capable the badge turns red before you waste a run on it, and the vision selector only offers models that can actually see.
python -m src.cli infoPrints vector-store stats for the collections under ./data/. For anything interactive — picking models, starting pipelines, browsing frames and artifacts — use the web command deck above; it is the primary GUI.
The hybrid change detector uses three complementary signals to decide when the screen has actually changed:
| Signal | What it detects | Threshold |
|---|---|---|
| SSIM | Pixel-level structural changes | < 0.97 |
| pHash | Perceptual content changes via DCT | hamming >= 8 |
| HSV histogram | Color-distribution shifts | correlation <= 0.90 |
A keyframe is emitted when any signal triggers and the minimum 0.5-second interval has elapsed. A forced keyframe every four seconds catches slow scrolling. This typically captures a small fraction of source frames while preserving distinct screens.
All settings live in src/config.py as Pydantic models. Key parameters:
| Parameter | Default | Description |
|---|---|---|
frame_extraction.strategy |
keyframe | Smart change detection or fixed_fps |
frame_extraction.max_interval_seconds |
4.0 | Maximum gap between keyframes |
captioning.backend |
platform-dependent | vllm, omlx, or optional ollama |
captioning.vllm_base_url |
http://127.0.0.1:8000/v1 | DGX Spark OpenAI-compatible API URL |
captioning.vllm_model |
null | Falls back to VLLM_MODEL, then the checked NVIDIA Qwen model |
captioning.omlx_base_url |
http://127.0.0.1:8000/v1 | Apple oMLX URL; root/dashboard URLs are normalized |
captioning.omlx_model |
null | Falls back to MLX_MODEL/OMLX_MODEL/LLM_MODEL, then default |
captioning.batch_size |
2 DGX / 4 Apple | Concurrent direct-server caption requests |
captioning.max_tokens |
32768 | Requested caption ceiling; vLLM uses the context remaining after image/prompt tokens |
captioning.retry_attempts |
1 | Per-frame retries after a direct caption request fails |
captioning.retry_max_tokens |
2048 | Bounded retry ceiling that prevents malformed generations from consuming another full caption budget |
captioning.repetition_penalty |
1.05 | Discourage pathological long caption loops |
captioning.no_repeat_ngram_size |
12 | Block repeated long caption sequences; 0 disables |
captioning.disable_thinking |
true | Spend the output budget on the visible answer |
embedding.model_name |
ViT-B-32 | OpenCLIP architecture |
embedding.device |
CUDA DGX / MPS Apple | Accelerator for OpenCLIP; CPU elsewhere |
ocr.backend |
platform-dependent | Direct vllm or omlx vision endpoint |
ocr.concurrency |
2 DGX / 4 Apple | Concurrent verbatim OCR requests |
reconstruction.backend |
platform-dependent | Direct text inference endpoint |
reconstruction.timeout_seconds |
1800 | Timeout for long-running reconstruction generations |
reconstruct gives every extraction pass the whole model context as its output
ceiling, so vLLM can compute the exact remaining completion space per prompt.
On Spark that is close to free. On oMLX it is not: a measured 20-caption run
(98k characters of captions) spent over three hours in Pass 1 alone, because
each of its three segments generated 20–30k tokens at 6–12 tok/s and two of
them exhausted the 32K ceiling and split into two passes each.
Before reaching for reconstruct on Apple Silicon, check whether summarize
is what you actually want — it chunks to a bounded output and finished the same
run in four minutes.
Direct backends receive one OpenAI-compatible image request per frame. Image encoding and prompt prefill usually dominate short captions, so the main levers are frame dimensions, model size, and request concurrency. The bundled Spark service admits two sequences and ScreenLens therefore defaults to two requests there. Apple defaults to four, but large oMLX models may benefit from a lower value. Concurrent results are isolated per frame: one failed request cannot overwrite a successful peer in the same chunk. A failed frame is retried once deterministically with a bounded 2K completion ceiling. A retry that fills that ceiling without terminating is rejected instead of storing a truncated loop; ScreenLens stores an error marker for that frame alone.
DGX Spark has one 128 GB unified-memory pool rather than separate system RAM and VRAM. The checked single-service recipe uses the dense NVFP4 27B model, a 0.45 vLLM allocator target, a 262K context, and concurrency two. Caption requests retain a 32K output ceiling, leaving the rest of the served window for prompt, chat-template, and image tokens. If a service is deliberately reduced to a matching 32K context, ScreenLens omits the literal output limit and lets vLLM allocate the exact context remaining after its input.
Reconstruction groups captions by serialized size rather than frame count, so an unusually long caption cannot overfill a later prompt. Its shared extraction pass requests dense notes under 1,400 tokens but gives the model the full context exposed by the server as completion headroom. Recursive synthesis filters notes to the file or artifact currently being rebuilt and uses that same full ceiling. These long text generations use the independent 1,800-second reconstruction.timeout_seconds budget instead of the shorter caption timeout. If a response still ends with finish_reason=length, ScreenLens discards the incomplete prefix and retries a smaller input group. The 262K service context is therefore used automatically when VLLM_MAX_MODEL_LEN matches it. Native two-token Qwen MTP accelerates decoding without changing answers; DFlash is not loaded because it would add a second draft model and does not increase context or output quality. See the Spark memory notes.
ffprobe remains optional: when it is absent, video metadata is read through OpenCV without a warning. OpenCLIP's checked checkpoint is public, so ScreenLens suppresses only the Hub's unauthenticated-download advisory; actual authorization, rate-limit, and download failures remain visible. A Hugging Face token is still required when the DGX helper must start the gated vLLM model service.
compose.dgx-spark.yaml # Bounded ARM64/CUDA vLLM service
docs/
DGX_SPARK.md # Setup, sizing, reuse, lifecycle, troubleshooting
setup_and_run_dgx.sh # Checked Spark setup, validation, and launcher
setup_and_run_macos.sh # Apple Silicon Conda/oMLX launcher
src/
cli.py # Stable Typer command entry point
config.py # Shared platform-aware Pydantic settings
session.py # Config loading, model roles, and run discovery
inference/
client.py # Provider-neutral Spark/oMLX client
captioner.py # Frame captioning
ocr.py # Verbatim vision OCR
video/
frame_extractor.py # Keyframe extraction
frame_select.py # Dense sampling for transcription
stitch.py # Scroll-overlap text stitching
storage/
embedder.py # OpenCLIP embeddings
vector_store.py # ChromaDB storage and search
workflows/
pipeline.py # Ingest/search/summarize graphs
transcribe.py # OCR and guarded cleanup
reconstruct.py # Artifact reconstruction and QA
assemble.py # Corpus assembly
web/
server.py # Loopback HTTP API and static assets
runner.py # Background job coordination
static/
index.html # Dashboard markup
deck.css # Nightshift-style presentation
deck.js # Dashboard interactions and API calls
favicon.svg
tests/
test_pipeline.py # Core configuration, inference, embedding, and graph tests
test_transcribe.py # Stitching, OCR guards, and cleanup safety
test_web.py # Command deck assets, API contracts, and security
test_cases.yaml # Dual-platform/DGX end-to-end scenarios
Generated frames, captions, databases, videos, outputs, virtual environments, and model caches are ignored by Git.
| Feature | NVIDIA VSS | ScreenLens |
|---|---|---|
| Frame extraction | Custom + TensorRT | Hybrid SSIM/pHash/HSV detection |
| Vision model | NVIDIA VILA | NVIDIA Qwen3.6 via vLLM or a configured oMLX model |
| Embeddings | TensorRT visual encoder | OpenCLIP ViT-B-32 |
| Vector DB | Milvus | ChromaDB |
| LLM | Llama 3.1 70B / NIM | Same local vLLM/oMLX endpoint used by the pipeline |
| Hardware | NVIDIA GPU | DGX Spark or Apple Silicon |
| Deployment | Docker + NIM | Bounded vLLM container + Python, or native oMLX + Python |
| Cloud dependency | None when self-hosted | None; local model endpoints only |
- Harden near-duplicate keyframe filtering (perceptual hash + SSIM fusion threshold tuning)
- Cross-video deduplication for multi-file ingestion
- Consider leveraging Karpathy's autoresearch to iterate on dedup thresholds against a fixed evaluation set
Preconfigured extraction and captioning strategies tailored to content type:
| Profile | Description | Audio | Typical Source |
|---|---|---|---|
code |
Silent screen recording of browsing/editing code | No | IDE walkthroughs, code reviews |
demo |
Screencast with voice-over demonstrating software | Yes | Product demos, tutorials, onboarding |
pdf |
Continuous scroll/browse of a PDF | No | Recorded read-throughs, slide decks |
meeting |
Video call or presentation | Yes | Zoom/Teams recordings, webinars |
Each profile will tune frame extraction, captioning prompts, chunking, and audio processing.
- Integrate Whisper speech-to-text via ONNX Runtime and/or MLX
- Support
small,medium, andlargemodel sizes - Add word-level timestamps aligned to the keyframe timeline
- Fuse captions and transcripts for richer semantic search
- Manual generator with extracted screenshots and navigation flow
- PDF summary preserving headings, key points, and figures
- Source-code reconstruction with files, signatures, and project structure
- Meeting notes with action items, decisions, and speaker attribution
Run screenlens serve to open the Nightshift-style command deck. Choose
Spark cluster · vLLM for http://sparkone.local:8000/v1, or
oMLX · Apple Silicon for http://127.0.0.1:8000/v1. These presets match
Nightshift; the endpoint field remains editable for other cluster addresses.
Endpoint checks and model lists follow the selected backend and URL, and jobs
use those same form values. Select a served vision model for captions/OCR and
a served text model for reconstruction. The deck supports only these two
inference backends; existing CLI compatibility is retained.
The Spark preset uses the cluster hostname so a changed LAN IP does not leave
it pointing at an old address. Qwen3.8 models (including the served alias
qwen3.8-flash-next) are recognized for captions/OCR. If an endpoint check
fails, correct the URL or restore the server, then click Retry connection.
An unreachable endpoint is reported separately from a reachable server with
no recognized vision models; switching endpoints clears the old model choices.
Shared configuration and session helpers stay at the package root. Inference
handles model requests, video handles frames and stitching, storage handles
embeddings and retrieval, and workflows coordinate those components. The CLI
and web runner call workflows. Internal imports now use these explicit package
paths; screenlens, python -m src.cli, and python -m src.web are unchanged.
Design history lives in docs/REDESIGN.md. Local data, environment backups,
agent state, and tool caches stay ignored and are not moved by this refactor.