vdisplay

AI Cost Tracking

🤖 LLM usage: $42.0547 (41 commits)
👤 Human dev: ~$2111 (21.1h @ $100/h, 30min dedup)

Generated on 2026-06-17 using openrouter/qwen/qwen3-coder-next

Cross-platform virtual display orchestration API for Python.

One unified API, multiple OS backends with different capabilities. Monitors and windows include an nl field — a natural-language description of what they contain.

CLI, DSL, REST, MCP, and the local vdisplay-agent broker all route through application.executor and shared application services (src/vdisplay/application/).

Quick start

pip install "vdisplay[pillow]"
# or from source (recommended for development):
pip install -e ".[pillow,dev]"
pip install -e "packages/vdisplay-agent[serve]"
pip install -e packages/dsl2vdisplay packages/rest2vdisplay packages/mcp2vdisplay

unset DISPLAY   # optional: auto-resolves host display to :0
vdisplay all
vdisplay monitors
vdisplay windows --apps-only

Documentation

Start	Description
docs/start-here.md	Entry point — install, local vs broker, choose your path

Guides	Reference
Desktop control today	Environment vars
Wayland control	CLI index
GUI Map Pack	API / SDK
Vision fallback	DSL · REST · MCP
Agent broker
Browser · Terminal

Architecture	Other
docs/architecture.md	Installation
Control plane	Examples index
Agent broker (full API)	Troubleshooting
Vision-only Wayland	packages/README.md
Electron Share Manager	Electron package README

Full index: docs/index.md

Examples

Example	Mode	Host X11	Run
headless-virtual	virtual	No	`cd examples/headless-virtual && docker compose up --build`
agent-broker	broker	No	`cd examples/agent-broker && ./run.sh`
ci-agent	virtual	No	`cd examples/ci-agent && docker compose run --rm ci-agent`
dev-workspace	dev	No	`cd examples/dev-workspace && docker compose run --rm dev`
host-mirror	mirror	Yes	`cd examples/host-mirror && ./run.sh`
host-relay	relay	Yes	`cd examples/host-relay && ./run.sh`
run_all_examples.sh	mixed	varies	`./examples/run_all_examples.sh`

Details: docs/examples.md · per-example READMEs in each folder above.

Use cases

Goal	Start here
Koru autonomous loop + photo-VQL chat drive	examples/dev-workflow/README.md · autonomy-loop.md
Desktop automation on GNOME Wayland	docs/guides/wayland-control.md
Persistent vision click targets	docs/guides/gui-map-pack.md
Broker + REST/MCP for agents	docs/guides/agent-broker.md
Stable Wayland capture with preview window	docs/electron-share-manager.md
Headless CI screenshot	examples/headless-virtual
Semantic browser/terminal control	docs/control-plane.md

Common commands

vdisplay all
vdisplay monitors
vdisplay windows --apps-only
vdisplay diagnose
vdisplay screenshot -o out.png --source DP-1    # Wayland: agent + screencast first
vdisplay diagnose control
vdisplay control click --backend vision --map maps/chat.json --target chat
vdisplay services up --install --instance pycharm --target "PyCharm chat" --source HDMI-1 --open-browser-bridge

CLI index: docs/reference/cli.md

IDE & Desktop App Control

Launch applications and prompt IDEs from the command line.

Application Launcher:

vdisplay app list
vdisplay app show pycharm
vdisplay app open pycharm
vdisplay app open pycharm --variant default-xwayland

End-to-end IDE Prompting:

vdisplay ide list
vdisplay ide prompt --ide pycharm --text "Explain this stack trace"
vdisplay ide prompt --ide pycharm --open --map maps/pycharm-chat.json --submit

Koru + photo-VQL (autonomous chat drive)

Full observe → decide → act → verify loop with session audit and capture guards (capture_confirmed):

cd examples/dev-workflow
bash koru-drive-photo-vql.sh --ide jetbrains --source DP-2 --prompt "fix tests" --submit
bash koru-audit-last-session.sh --ide jetbrains

Recommended first step on GNOME Wayland:

vdisplay services up \
  --install \
  --instance jetbrains \
  --target jetbrains \
  --source HDMI-1 \
  --open-browser-bridge

Then in the opened browser bridge tab click Share screen, select the IDE monitor, and keep the tab open. Koru should only run photo-VQL drive after capture_ready=true; otherwise it may fall back to blind keyboard injection.

Requires: KORU_SRC, IMGL_SRC, venv with [observe].
Guide: examples/dev-workflow/README.md.

See Desktop control today for implementation details and limitations.

Output objects (`nl`)

Monitors and windows include nl — a natural-language summary of contents.

vdisplay all | jq '{monitors: .monitors[].nl, windows: .windows[].nl}'

Desktop workflows

Long-form GNOME / JetBrains / Firefox / screenshot workflows moved to guides:

Legacy detail: extended README sections (desktop workflows, control plane examples, screenshot recipes) remain in git history; prefer guides for maintenance.

Control plane

Unified AT-SPI, browser, terminal, vision, and map backends. See docs/control-plane.md and task guides above.

vdisplay diagnose control
vdisplay control list --backend auto
vdisplay control click --role button --name Save --verify

vdisplay-agent broker

# Terminal 1 — same GNOME session
export PYTHONPATH=src:packages/vdisplay-agent/src
vdisplay-agent serve --port 8765

# Terminal 2
export VDISPLAY_AGENT_URL=http://127.0.0.1:8765
export PYTHONPATH=src:packages/vdisplay-agent/src
vdisplay agent preflight
vdisplay agent screencast start --force   # portal → All Screens
vdisplay screenshot -o /tmp/host.png --source DP-1

GNOME Wayland 3-monitor: docs/guides/gnome-wayland-screencast.md
Dev automation on this PC: examples/dev-workflow/

Full API: docs/agent-broker.md · Guide: docs/guides/agent-broker.md

Electron Share Manager (Wayland capture window)

For GNOME Wayland hosts where Python PipeWire capture is unreliable, use the orchestrated services command. It starts vdisplay-agent, starts the Electron manager UI/tray, and exposes a Chrome/Chromium browser bridge that pushes PNG frames to the agent.

One-command first run:

cd ~/github/wronai/vdisplay
source .venv/bin/activate

vdisplay services up --install \
  --instance pycharm \
  --target "PyCharm chat" \
  --source HDMI-1 \
  --open-browser-bridge

Then approve sharing in the opened browser tab:

Click Share screen.
Select the IDE monitor.
Keep the tab open while automation runs.

After capture_ready=true, capture can read frames from the agent:

export VDISPLAY_AGENT_URL=http://127.0.0.1:8766
vdisplay screenshot -o /tmp/pycharm.png --source HDMI-1

The lower-level Electron manager can still be launched directly when you only need the tray/full-window manager:

vdisplay electron-share up --install \
  --instance pycharm \
  --target "PyCharm chat" \
  --source HDMI-1 \
  --port 8799

vdisplay electron-share start is the foreground/debug variant; use vdisplay electron-share up or vdisplay services up for a manager that survives after the CLI returns.

Docs: docs/electron-share-manager.md · Package: packages/vdisplay-electron-share/README.md

Control layer equivalents

Intent	CLI	DSL
Full state	`vdisplay all`	`ALL DISPLAY :0`
Monitors	`vdisplay monitors`	`MONITORS DISPLAY :0`
Windows	`vdisplay windows --apps-only`	`WINDOWS DISPLAY :0`
Adopt window	`vdisplay relay adopt-window --app X`	`ADOPT APP X`
Screenshot	`vdisplay screenshot -o out.png`	`SCREENSHOT OUT out.png DISPLAY :99`
Validate tools	`vdisplay diagnose`	`VALIDATE DISPLAY :0`

Modes

Mode	Purpose	Isolation	Screenshot	Window move
`virtual`	Private Xvfb session for agents	Yes	Yes	No (use `launch()`)
`mirror`	Duplicate existing display output	No	Yes	N/A
`relay`	Move window within same X11 session	Partial	Yes (`relay screenshot`)	Yes
`screencast`	Portal ScreenCast in agent (Wayland)	No	Yes (after consent)	N/A

Requirements (Linux v0.1)

Component	Used by
`Xvfb`, `xwd`, `scrot`	virtual / X11 capture
`xrandr`	mirror mode
`xdotool`	relay + input
`python3-dbus`, `python3-gi`	portal ScreenCast (Wayland host)
`ffmpeg` (PipeWire) or GStreamer `pipewiresrc`	ScreenCast frame grab
`Pillow` (optional)	faster PNG encoding

sudo apt install xvfb x11-apps x11-utils xdotool scrot x11-xserver-utils
sudo apt install python3-dbus python3-gi   # Wayland ScreenCast in agent
pip install "vdisplay[pillow]"

Full setup: docs/installation.md

Python API

Sessions (backends)

from vdisplay import VirtualDisplaySession, MirrorSession, WindowRelaySession
from vdisplay.discovery import list_monitors, list_windows

# Inspect monitors and windows with nl descriptions
for monitor in list_monitors():
    print(monitor["nl"])

for window in list_windows(apps_only=True):
    print(window["nl"])

# Virtual isolated display
vd = VirtualDisplaySession.create(width=1920, height=1080)
vd.start()
vd.launch(["xterm"])
vd.save_screenshot("screen.png")
vd.stop()

# Mirror existing desktop (same session, no isolation)
# On GNOME Wayland: start agent screencast before save_screenshot, or use capture_host_to_file()
m = MirrorSession.create(source="primary", target="DP-1")
m.start()
m.save_screenshot("mirror.png")   # needs ScreenCast on Wayland
m.stop()

# Relay window off-screen and restore (persists across CLI calls)
r = WindowRelaySession.create()
r.start()
r.adopt_window(match_app="JetBrains")
r.release_window(match_app="JetBrains")
r.stop()

Application layer (shared by CLI / DSL / REST / MCP / agent)

from vdisplay.application.commands import CommandRequest
from vdisplay.application.executor import execute
from vdisplay.application.services import discovery, capture, session, info

# Single execution entry (routes to agent when VDISPLAY_AGENT_URL is set)
result = execute(CommandRequest.from_dsl({"verb": "MONITORS"}, line="MONITORS"))
print(result.data)

# Direct service use-cases (always in-process)
monitors = discovery.list_monitors(display=":0")
meta = capture.capture_screenshot(output="screen.png", monitor=1)
session.virtual_start(width=1280, height=720, display=":99")
caps = info.platform_info()

Via broker SDK:

from vdisplay.client import AgentClient

client = AgentClient("http://127.0.0.1:8765")
client.outputs()
client.start_screencast(interactive=True)

Window heuristics (testable submodules)

from vdisplay.windows import list_windows_enriched, find_windows, pick_best_window
from vdisplay.windows.rank import dedupe_app_windows
from vdisplay.windows.filter import is_internal_window

Project layout

src/vdisplay/
├── application/
│   ├── commands.py         # CommandRequest / CommandResult / CommandVerb
│   ├── executor.py         # single entry: execute() → local or agent
│   ├── handlers/           # local + agent command handlers
│   └── services/           # discovery, capture, session, info
├── commands/               # CLI registry (set_defaults per subcommand)
├── windows/                # scan → normalize → filter → rank → query
├── capture/
│   ├── providers/          # drm, fbdev, mss, x11, portal (opt-in)
│   └── portal_screencast.py  # persistent ScreenCast (Wayland)
├── backends/               # virtual, mirror, relay
├── client.py               # AgentClient SDK
└── cli.py                  # thin entry: register_all + args.func(args)

packages/
├── dsl2vdisplay/           # grammar + CQRS bus → executor
├── vdisplay-agent/         # localhost broker (privileged runtime)
├── rest2vdisplay/          # HTTP → DSL
├── mcp2vdisplay/           # MCP tools
└── nlp2vdisplay/           # NL → DSL

Control layer (DSL / MCP / REST / NL)

Programmatic interfaces on top of the same application services. All query results include nl on monitors and windows.

Package	Role
dsl2vdisplay	Grammar + CQRS bus (`MONITORS`, `WINDOWS`, `ADOPT`, …)
nlp2vdisplay	Natural language → DSL
uri2vdisplay	`vdisplay://cmd/...` → DSL
cli2vdisplay	REPL over DSL
mcp2vdisplay	MCP server tools
rest2vdisplay	REST API on port 8216
vdisplay-agent	Local broker — sessions, capture, relay

vdisplay-agent serve --port 8765
export VDISPLAY_AGENT_URL=http://127.0.0.1:8765

pip install -e packages/dsl2vdisplay packages/rest2vdisplay packages/mcp2vdisplay
rest2vdisplay serve --port 8216 --agent-url $VDISPLAY_AGENT_URL
mcp2vdisplay serve
curl -s http://127.0.0.1:8216/health | jq .
curl -s -X POST http://127.0.0.1:8216/v1/dsl -H 'content-type: text/plain' -d 'MONITORS' | jq .

Full reference: packages/README.md

Limitations

Existing windows on DISPLAY=:0 cannot move into Xvfb :99 — different X servers.
Use VirtualDisplaySession.launch() for apps on the virtual display.
Use WindowRelaySession to hide/show windows on the current session.
mirror controls the same desktop through a duplicated output, not an isolated copy.
nl on monitors lists apps whose window center falls on that output geometry.
On GNOME Wayland, Docker X11 forwarding often produces black screenshots — use vdisplay-agent on the host instead.
Windows/macOS backends are planned; Linux/X11 + Wayland (via agent) supported in v0.1.

Troubleshooting: docs/troubleshooting.md

Development

pip install -e ".[pillow,dev]"
pip install -e "packages/vdisplay-agent[serve]"
pip install -e packages/dsl2vdisplay packages/rest2vdisplay packages/mcp2vdisplay
pytest tests/ -q
./examples/agent-broker/run.sh
./examples/run_all_examples.sh   # where host X11 is available

Architecture: docs/architecture.md

License

Licensed under Apache-2.0.

Name		Name	Last commit message	Last commit date
Latest commit History 49 Commits
.cursor		.cursor
.github/workflows		.github/workflows
.koru		.koru
.planfile		.planfile
brain		brain
docs		docs
examples		examples
maps		maps
packages		packages
packaging/systemd		packaging/systemd
project		project
proto/vdisplay/v1		proto/vdisplay/v1
src/vdisplay		src/vdisplay
testql-scenarios		testql-scenarios
tests		tests
.gitignore		.gitignore
.nojekyll		.nojekyll
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
README.md		README.md
SUMD.md		SUMD.md
SUMR.md		SUMR.md
TODO.md		TODO.md
VERSION		VERSION
app.doql.less		app.doql.less
app.render.png		app.render.png
app.vql.json		app.vql.json
app.vql.render.png		app.vql.render.png
goal.yaml		goal.yaml
host.png		host.png
index.html		index.html
koru.yaml		koru.yaml
mirror.png		mirror.png
nlp2uri.yaml		nlp2uri.yaml
planfile.yaml		planfile.yaml
prefact.yaml		prefact.yaml
project.sh		project.sh
pycharm-test.png		pycharm-test.png
pyproject.toml		pyproject.toml
relay.png		relay.png
screen.png		screen.png
screen.png.context.json		screen.png.context.json
screen.png.vql.imgl.json		screen.png.vql.imgl.json
screen.png.vql.json		screen.png.vql.json
test-screencast.png		test-screencast.png
toolbox-test.png		toolbox-test.png
tree.sh		tree.sh
tree.txt		tree.txt
ui.vql.json		ui.vql.json
vd.png		vd.png
vdisplay.yaml		vdisplay.yaml
virtual-test.png		virtual-test.png

Navigation Menu

Uh oh!

Repository files navigation

vdisplay

AI Cost Tracking

Quick start

Documentation

Examples

Use cases

Common commands

IDE & Desktop App Control

Koru + photo-VQL (autonomous chat drive)

Output objects (`nl`)

Desktop workflows

Control plane

vdisplay-agent broker

Electron Share Manager (Wayland capture window)

Control layer equivalents

Modes

Requirements (Linux v0.1)

Python API

Sessions (backends)

Application layer (shared by CLI / DSL / REST / MCP / agent)

Window heuristics (testable submodules)

Project layout

Control layer (DSL / MCP / REST / NL)

Limitations

Development

License

About

Releases

Packages

Contributors

Languages

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

vdisplay

AI Cost Tracking

Quick start

Documentation

Examples

Use cases

Common commands

IDE & Desktop App Control

Koru + photo-VQL (autonomous chat drive)

Output objects (nl)

Desktop workflows

Control plane

vdisplay-agent broker

Electron Share Manager (Wayland capture window)

Control layer equivalents

Modes

Requirements (Linux v0.1)

Python API

Sessions (backends)

Application layer (shared by CLI / DSL / REST / MCP / agent)

Window heuristics (testable submodules)

Project layout

Control layer (DSL / MCP / REST / NL)

Limitations

Development

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Output objects (`nl`)